Jul 24 12:18:44.970357 ip-10-0-140-61 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 12:18:45.425796 ip-10-0-140-61 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 12:18:45.425796 ip-10-0-140-61 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 12:18:45.425796 ip-10-0-140-61 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 12:18:45.425796 ip-10-0-140-61 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 12:18:45.425796 ip-10-0-140-61 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 12:18:45.428417 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.428295    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 12:18:45.435772 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435753    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:18:45.435772 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435770    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:18:45.435772 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435774    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:18:45.435772 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435778    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435781    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435784    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435787    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435790    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435793    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435796    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435799    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435802    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435805    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435807    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435812    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435816    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435819    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435822    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435825    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435828    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435831    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435834    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:18:45.435922 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435836    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435844    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435847    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435850    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435852    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435855    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435859    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435862    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435864    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435867    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435869    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435872    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435874    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435877    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435879    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435883    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435886    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435888    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435891    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435894    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:18:45.436410 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435896    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435899    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435901    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435903    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435906    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435908    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435911    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435913    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435917    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435921    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435924    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435927    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435930    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435933    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435936    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435939    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435941    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435943    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435946    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435974    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:18:45.436879 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435977    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435980    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435984    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435987    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435989    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435994    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.435997    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436001    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436003    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436006    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436009    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436011    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436015    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436018    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436020    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436022    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436025    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436027    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436030    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436032    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:18:45.437376 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436034    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436037    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436039    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436042    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436044    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436046    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436049    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436052    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436055    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436057    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436060    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436062    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436065    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436067    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436070    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436072    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436090    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436804    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436812    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:18:45.437859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436815    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436818    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436821    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436823    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436826    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436829    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436832    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436835    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436837    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436841    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436844    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436849    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436853    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436856    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436860    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436863    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436866    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436869    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436872    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:18:45.438341 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436875    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436878    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436881    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436884    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436887    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436890    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436892    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436895    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436897    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436900    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436902    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436905    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436908    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436911    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436913    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436916    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436918    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436921    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436923    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:18:45.438821 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436926    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436929    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436932    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436934    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436937    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436940    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436942    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436944    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436947    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436949    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436952    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436954    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436956    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436959    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436961    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436963    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436966    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436969    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436971    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436974    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:18:45.439306 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436976    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436978    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436980    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436983    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436985    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436989    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436993    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436995    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.436998    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437000    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437003    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437005    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437007    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437010    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437013    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437016    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437018    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437021    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437023    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437026    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:18:45.439789 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437029    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437032    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437034    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437036    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437039    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437042    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437044    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437046    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437049    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437052    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437055    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437058    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437060    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437062    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437065    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437067    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437070    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437090    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437093    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437489    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:18:45.440302 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437494    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437497    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437500    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437503    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437505    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437508    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437510    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437513    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437515    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437518    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437520    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437523    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437525    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437527    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437530    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437532    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437535    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437537    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437539    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437541    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:18:45.440877 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437544    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437547    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437549    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437552    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437554    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437557    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437559    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437562    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437564    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437569    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437573    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437577    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437580    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437583    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437586    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437590    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437593    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437596    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437598    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:18:45.441397 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437601    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437603    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437606    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437608    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437611    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437613    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437616    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437618    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437620    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437624    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437626    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437629    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437631    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437634    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437637    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437640    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437642    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437645    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437647    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437649    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:18:45.441863 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437652    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437654    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437658    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437661    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437664    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437666    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437679    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437683    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437685    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437688    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437690    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437693    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437695    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437698    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437701    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437705    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437708    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437711    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437713    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:18:45.442371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437716    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437719    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437721    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437724    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437726    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437729    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437731    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437734    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437736    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437739    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437741    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437743    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437746    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437748    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437751    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437754    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437757    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437759    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437762    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.437764    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:18:45.442896 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437828    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437835    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437841    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437845    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437849    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437852    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437856    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437860    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437864    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437867    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437870    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437874    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437877    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437880    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437883    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437886    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437889    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437892    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437896    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437899    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437902    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437905    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437908    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437912    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 12:18:45.443409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437915    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437918    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437921    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437925    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437928    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437931    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437933    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437936    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437940    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437943    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437946    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437948    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437953    2566 flags.go:64] FLAG: --enable-server="true"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437956    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437960    2566 flags.go:64] FLAG: --event-burst="100"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437963    2566 flags.go:64] FLAG: --event-qps="50"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437966    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437969    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437971    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437975    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437978    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437980    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437983    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437986    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437989    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 12:18:45.444006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437991    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437994    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437997    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.437999    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438002    2566 flags.go:64] FLAG: --feature-gates=""
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438006    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438009    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438012    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438015    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438018    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438022    2566 flags.go:64] FLAG: --help="false"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438025    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438028    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438031    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438033    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438037    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438040    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438042    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438045    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438050    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438052    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438055    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438058    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438061    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 24 12:18:45.444659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438064    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438066    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438069    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438072    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438090    2566 flags.go:64] FLAG: --lock-file=""
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438093    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438095    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438098    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438104    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438106    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438109    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438112    2566 flags.go:64] FLAG: --logging-format="text"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438115    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438118    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438121    2566 flags.go:64] FLAG: --manifest-url=""
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438124    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438128    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438131    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438136    2566 flags.go:64] FLAG: --max-pods="110"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438138    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438141    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438144    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438147    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438150    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438153    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 12:18:45.445320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438155    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438162    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438166    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438169    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438171    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438174    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438179    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438182    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438185    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438188    2566 flags.go:64] FLAG: --port="10250"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438190    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438193    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-086e621d2ef898917"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438196    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438199    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438202    2566 flags.go:64] FLAG: --register-node="true"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438205    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438208    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438211    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438214    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438216    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438220    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438223    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438226    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438229    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438232    2566 flags.go:64] FLAG: --runonce="false"
Jul 24 12:18:45.445912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438234    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438238    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438241    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438244    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438246    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438249    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438252    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438255    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438258    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438261    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438264    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438267    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438270    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438273    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438276    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438281    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438284    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438286    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438290    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438293    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438296    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438298    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438301    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438304    2566 flags.go:64] FLAG: --v="2"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438308    2566 flags.go:64] FLAG: --version="false"
Jul 24 12:18:45.446540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438312    2566 flags.go:64] FLAG: --vmodule=""
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438315    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.438318    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438432    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438435    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438439    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438442    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438445    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438448    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438451    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438453    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438456    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438459    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438461    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438464    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438467    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438470    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438473    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438475    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438478    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:18:45.447226 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438480    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438483    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438485    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438487    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438490    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438492    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438495    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438497    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438500    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438503    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438505    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438508    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438510    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438513    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438516    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438518    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438520    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438524    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438527    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:18:45.447774 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438531    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438534    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438537    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438540    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438543    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438546    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438549    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438551    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438554    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438557    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438559    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438562    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438566    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438569    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438572    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438574    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438577    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438580    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438582    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:18:45.448551 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438585    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438588    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438590    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438592    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438595    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438597    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438599    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438602    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438604    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438607    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438609    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438612    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438614    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438617    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438621    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438623    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438626    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438628    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438631    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438633    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:18:45.449067 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438636    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438638    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438642    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438644    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438646    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438649    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438651    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438654    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438657    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438659    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438661    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438664    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438666    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438669    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438671    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438674    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438678    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438681    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438683    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438686    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:18:45.449593 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438688    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438691    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438693    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.438696    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.439361    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.446599    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.446616    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.446636    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.446641    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446693    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446712    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446727    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446730    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446733    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:18:45.450071 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446737    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446740    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446743    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446745    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446748    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446751    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446754    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446756    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446759    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446761    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446764    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446766    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446769    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446771    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446774    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446776    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446779    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446781    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446784    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:18:45.450444 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446786    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446788    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446791    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446795    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446798    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446802    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446806    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446809    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446812    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446815    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446818    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446821    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446825    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446828    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446831    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446833    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446836    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446838    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446841    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:18:45.450909 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446843    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446846    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446848    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446850    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446853    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446855    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446857    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446860    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446862    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446865    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446867    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446869    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446872    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446874    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446876    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446879    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446881    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446883    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446886    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446889    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446892    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:18:45.451411 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446895    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446898    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446900    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446903    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446906    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446908    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446911    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446913    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446915    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446918    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446920    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446923    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446925    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446927    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446930    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446933    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446935    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446937    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446940    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446942    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:18:45.451957 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446945    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446947    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446949    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446952    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446954    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446956    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446959    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446961    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446963    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446967    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446970    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446972    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446975    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446977    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.446980    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:18:45.452471 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.446985    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447124    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447128    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447131    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447134    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447137    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447139    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447142    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447144    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447147    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447149    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447151    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447154    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447156    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447159    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447161    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447165    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447168    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447171    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447173    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:18:45.452843 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447176    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447178    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447181    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447183    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447185    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447188    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447194    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447197    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447199    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447202    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447204    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447207    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447209    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447212    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447214    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447216    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447219    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447222    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447224    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:18:45.453416 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447227    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447230    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447232    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447235    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447237    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447240    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447242    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447245    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447247    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447251    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447254    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447257    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447260    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447262    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447264    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447267    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447269    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447271    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447274    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447276    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:18:45.453889 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447279    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447282    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447284    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447286    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447289    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447292    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447294    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447297    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447299    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447301    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447304    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447306    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447309    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447311    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447314    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447316    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447318    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447321    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447323    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:18:45.454379 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447325    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447329    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447332    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447335    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447338    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447340    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447343    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447345    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447348    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447350    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447352    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447355    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447357    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447360    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447362    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447365    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447367    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447370    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447373    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447375    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:18:45.454859 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447378    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:18:45.455347 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:45.447380    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:18:45.455347 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.447384    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 12:18:45.455347 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.447484    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 12:18:45.455347 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.451184    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 12:18:45.455347 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.453760    2566 server.go:1019] "Starting client certificate rotation"
Jul 24 12:18:45.455347 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.453857    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 12:18:45.455347 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.453896    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 12:18:45.457428 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.457413    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 12:18:45.458629 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.458615    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 12:18:45.467662 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.467647    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 24 12:18:45.469944 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.469932    2566 log.go:25] "Validated CRI v1 image API"
Jul 24 12:18:45.470831 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.470813    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 12:18:45.473497 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.473472    2566 fs.go:135] Filesystem UUIDs: map[69b9489f-8002-4f7c-b330-29c544a65b55:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b08a53be-5caa-4d45-a1f7-a0c832411510:/dev/nvme0n1p4]
Jul 24 12:18:45.473566 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.473496    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 12:18:45.479341 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.479222    2566 manager.go:216] Machine: {Timestamp:2026-07-24 12:18:45.477895206 +0000 UTC m=+0.378583358 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098756 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26065114222583eed3cf0fd907cbcb SystemUUID:ec260651-1422-2583-eed3-cf0fd907cbcb BootID:ac749287-7708-4ef5-b8d1-a6c0954f65da Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ec:91:a3:cf:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ec:91:a3:cf:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:f6:d2:2f:0a:9b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 12:18:45.479341 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.479337    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 12:18:45.479464 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.479452    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 12:18:45.482801 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.482780    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 12:18:45.482924 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.482803    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-61.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 12:18:45.483677 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.483664    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 12:18:45.483711 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.483681    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 12:18:45.483711 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.483688    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 12:18:45.484284 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.484274    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 12:18:45.485494 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.485481    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 12:18:45.485567 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.485479    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 12:18:45.487345 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.487333    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 12:18:45.487465 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.487455    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 12:18:45.487500 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.487485    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 12:18:45.487500 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.487496    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 12:18:45.487553 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.487509    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 12:18:45.487553 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.487518    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 24 12:18:45.487553 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.487526    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 12:18:45.487875 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.487854    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 12:18:45.487875 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.487877    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 12:18:45.491707 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.491662    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 12:18:45.494300 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.494274    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 12:18:45.494379 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.494313    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 12:18:45.496275 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.496261    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 12:18:45.496360 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.496282    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 12:18:45.496360 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.496291    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 12:18:45.496360 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.496299    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 12:18:45.496360 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.496312    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 12:18:45.496360 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.496320    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 12:18:45.496360 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.496331    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 12:18:45.496360 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.496339    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 12:18:45.496360 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.496349    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 12:18:45.496360 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.496357    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 12:18:45.496620 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.496369    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 12:18:45.496620 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.496377    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 12:18:45.497258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.497247    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 12:18:45.497313 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.497261    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 12:18:45.499051 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.499018    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 12:18:45.499051 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.499018    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 12:18:45.500480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.500464    2566 server.go:1268] "Started kubelet"
Jul 24 12:18:45.500558 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.500532    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 12:18:45.501186 ip-10-0-140-61 systemd[1]: Started Kubernetes Kubelet.
Jul 24 12:18:45.501301 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.501249    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 12:18:45.501341 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.501315    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 12:18:45.502689 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.502674    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 12:18:45.505245 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.505226    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 12:18:45.507734 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.507703    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 12:18:45.508037 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.508025    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-61.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 12:18:45.509425 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.508349    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-61.ec2.internal.18c539060ea34a37  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-61.ec2.internal,UID:ip-10-0-140-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-61.ec2.internal,},FirstTimestamp:2026-07-24 12:18:45.500439095 +0000 UTC m=+0.401127247,LastTimestamp:2026-07-24 12:18:45.500439095 +0000 UTC m=+0.401127247,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-61.ec2.internal,}"
Jul 24 12:18:45.514658 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.514645    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 12:18:45.514731 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.514655    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 12:18:45.514731 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.514724    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:45.515407 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.515392    2566 factory.go:55] Registering systemd factory
Jul 24 12:18:45.515489 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.515411    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 24 12:18:45.515552 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.515529    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 12:18:45.515552 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.515545    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 12:18:45.515648 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.515578    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 12:18:45.515648 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.515619    2566 factory.go:153] Registering CRI-O factory
Jul 24 12:18:45.515735 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.515651    2566 factory.go:223] Registration of the crio container factory successfully
Jul 24 12:18:45.515735 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.515701    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 12:18:45.515820 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.515720    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 12:18:45.515820 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.515762    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 12:18:45.515820 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.515748    2566 factory.go:103] Registering Raw factory
Jul 24 12:18:45.515820 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.515804    2566 manager.go:1196] Started watching for new ooms in manager
Jul 24 12:18:45.516208 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.516195    2566 manager.go:349] Starting recovery of all containers
Jul 24 12:18:45.517308 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.517279    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 12:18:45.517627 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.517441    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 12:18:45.528343 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.528322    2566 manager.go:354] Recovery completed
Jul 24 12:18:45.530383 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.530362    2566 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 24 12:18:45.533385 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.533372    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:18:45.535172 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.535142    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:18:45.535258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.535185    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:18:45.535258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.535199    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:18:45.535758 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.535745    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 12:18:45.535758 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.535756    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 12:18:45.535856 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.535774    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 12:18:45.535856 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.535785    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:18:45.536445 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.536432    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:18:45.536487 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.536457    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:18:45.536487 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.536466    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:18:45.537513 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.537502    2566 policy_none.go:49] "None policy: Start"
Jul 24 12:18:45.537552 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.537519    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 12:18:45.537552 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.537529    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 12:18:45.538632 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.538622    2566 policy_none.go:47] "Start"
Jul 24 12:18:45.544441 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.544375    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-61.ec2.internal.18c5390610b53fac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-61.ec2.internal,UID:ip-10-0-140-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-61.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-61.ec2.internal,},FirstTimestamp:2026-07-24 12:18:45.535170476 +0000 UTC m=+0.435858629,LastTimestamp:2026-07-24 12:18:45.535170476 +0000 UTC m=+0.435858629,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-61.ec2.internal,}"
Jul 24 12:18:45.547743 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.547685    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-61.ec2.internal.18c5390610b58ceb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-61.ec2.internal,UID:ip-10-0-140-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-61.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-61.ec2.internal,},FirstTimestamp:2026-07-24 12:18:45.535190251 +0000 UTC m=+0.435878405,LastTimestamp:2026-07-24 12:18:45.535190251 +0000 UTC m=+0.435878405,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-61.ec2.internal,}"
Jul 24 12:18:45.549475 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.549458    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8xr94"
Jul 24 12:18:45.555342 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.555283    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-61.ec2.internal.18c5390610b5c5e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-61.ec2.internal,UID:ip-10-0-140-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-61.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-61.ec2.internal,},FirstTimestamp:2026-07-24 12:18:45.535204837 +0000 UTC m=+0.435892988,LastTimestamp:2026-07-24 12:18:45.535204837 +0000 UTC m=+0.435892988,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-61.ec2.internal,}"
Jul 24 12:18:45.557880 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.557824    2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-61.ec2.internal.18c5390610b53fac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-61.ec2.internal.18c5390610b53fac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-61.ec2.internal,UID:ip-10-0-140-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-61.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-61.ec2.internal,},FirstTimestamp:2026-07-24 12:18:45.535170476 +0000 UTC m=+0.435858629,LastTimestamp:2026-07-24 12:18:45.536445249 +0000 UTC m=+0.437133399,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-61.ec2.internal,}"
Jul 24 12:18:45.559242 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.559193    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8xr94"
Jul 24 12:18:45.581824 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.581701    2566 manager.go:337] "Starting Device Plugin manager"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.581859    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.581875    2566 server.go:87] "Starting device plugin registration server"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.582104    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.582113    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.582242    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.582345    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.582354    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.582863    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.582909    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.594132    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.595452    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.595486    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.595504    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.595537    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 12:18:45.602562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.599516    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:18:45.682955 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.682886    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:18:45.683817 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.683800    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:18:45.683886 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.683832    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:18:45.683886 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.683843    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:18:45.683886 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.683866    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.693641 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.693626    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.693707 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.693649    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-61.ec2.internal\": node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:45.695834 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.695818    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-61.ec2.internal"]
Jul 24 12:18:45.695896 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.695887    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:18:45.696618 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.696605    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:18:45.696671 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.696629    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:18:45.696671 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.696640    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:18:45.697901 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.697890    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:18:45.698027 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.698010    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.698065 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.698037    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:18:45.698569 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.698555    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:18:45.698616 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.698555    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:18:45.698616 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.698610    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:18:45.698676 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.698622    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:18:45.698676 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.698586    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:18:45.698676 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.698652    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:18:45.699873 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.699857    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.699946 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.699887    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:18:45.700544 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.700528    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:18:45.700612 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.700560    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:18:45.700612 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.700575    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:18:45.716973 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.716951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/017919978be3c1d1d7d67e195198972d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-61.ec2.internal\" (UID: \"017919978be3c1d1d7d67e195198972d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.717057 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.716975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f20104cb88cafd62fcd060f78a1481bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal\" (UID: \"f20104cb88cafd62fcd060f78a1481bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.717057 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.716992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f20104cb88cafd62fcd060f78a1481bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal\" (UID: \"f20104cb88cafd62fcd060f78a1481bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.717646 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.717633    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-61.ec2.internal\" not found" node="ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.720913 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.720898    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-61.ec2.internal\" not found" node="ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.733314 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.733291    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:45.817208 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.817182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f20104cb88cafd62fcd060f78a1481bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal\" (UID: \"f20104cb88cafd62fcd060f78a1481bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.817208 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.817211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/017919978be3c1d1d7d67e195198972d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-61.ec2.internal\" (UID: \"017919978be3c1d1d7d67e195198972d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.817381 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.817263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f20104cb88cafd62fcd060f78a1481bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal\" (UID: \"f20104cb88cafd62fcd060f78a1481bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.817381 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.817280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f20104cb88cafd62fcd060f78a1481bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal\" (UID: \"f20104cb88cafd62fcd060f78a1481bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.817381 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.817291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f20104cb88cafd62fcd060f78a1481bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal\" (UID: \"f20104cb88cafd62fcd060f78a1481bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.817381 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:45.817323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/017919978be3c1d1d7d67e195198972d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-61.ec2.internal\" (UID: \"017919978be3c1d1d7d67e195198972d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:45.834000 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.833977    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:45.934563 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:45.934497    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:46.021356 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.021317    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:46.025955 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.025742    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:46.035581 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:46.035564    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:46.136111 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:46.136063    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:46.236710 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:46.236630    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:46.337146 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:46.337117    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:46.437697 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:46.437662    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:46.452839 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.452811    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 12:18:46.452990 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.452975    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 12:18:46.537840 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:46.537775    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:46.564975 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.564932    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 12:13:46 +0000 UTC" deadline="2028-03-07 17:09:18.214666504 +0000 UTC"
Jul 24 12:18:46.564975 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.564965    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14212h50m31.649705247s"
Jul 24 12:18:46.566041 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.566025    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:18:46.567492 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.567475    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:18:46.567587 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.567506    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:18:46.567587 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.567516    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:18:46.567778 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:46.567748    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017919978be3c1d1d7d67e195198972d.slice/crio-bca565fc90a76a18e6cab246ce6cd57439d111647433f61f6ab5530348e40437 WatchSource:0}: Error finding container bca565fc90a76a18e6cab246ce6cd57439d111647433f61f6ab5530348e40437: Status 404 returned error can't find the container with id bca565fc90a76a18e6cab246ce6cd57439d111647433f61f6ab5530348e40437
Jul 24 12:18:46.572039 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.572023    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 12:18:46.574907 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.574890    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:18:46.599305 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.599269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-61.ec2.internal" event={"ID":"017919978be3c1d1d7d67e195198972d","Type":"ContainerStarted","Data":"bca565fc90a76a18e6cab246ce6cd57439d111647433f61f6ab5530348e40437"}
Jul 24 12:18:46.638853 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:46.638811    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-61.ec2.internal\" not found"
Jul 24 12:18:46.717259 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.717233    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:18:46.815201 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.815135    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:46.840420 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.840274    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 12:18:46.841899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.841711    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-61.ec2.internal"
Jul 24 12:18:46.854000 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.853985    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 12:18:46.995489 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:46.995461    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:18:48.861590 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.861564    2566 apiserver.go:52] "Watching apiserver"
Jul 24 12:18:48.873220 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.873191    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 12:18:48.873595 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.873574    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-44h9q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt","openshift-dns/node-resolver-hjxk2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal","openshift-multus/multus-m4x75","openshift-multus/network-metrics-daemon-zdv26","kube-system/global-pull-secret-syncer-gnrrg","kube-system/kube-apiserver-proxy-ip-10-0-140-61.ec2.internal","openshift-cluster-node-tuning-operator/tuned-8j47g","openshift-image-registry/node-ca-n6d7j","openshift-multus/multus-additional-cni-plugins-mm5x2","openshift-network-diagnostics/network-check-target-xmjh2","openshift-network-operator/iptables-alerter-bq9b6","openshift-ovn-kubernetes/ovnkube-node-4ccs4"]
Jul 24 12:18:48.875197 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.875178    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n6d7j"
Jul 24 12:18:48.876874 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.876856    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bq9b6"
Jul 24 12:18:48.878102 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.878083    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hjxk2"
Jul 24 12:18:48.878202 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.878153    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 12:18:48.878290 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.878270    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tbqh6\""
Jul 24 12:18:48.878540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.878521    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 12:18:48.878632 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.878617    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 12:18:48.879528 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.879505    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:18:48.879756 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.879736    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 12:18:48.879854 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.879769    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qldps\""
Jul 24 12:18:48.879854 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.879772    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.880800 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.880778    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 12:18:48.880800 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.880796    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 12:18:48.881135 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.881119    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 12:18:48.881215 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.881195    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pmrxk\""
Jul 24 12:18:48.881215 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.881200    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.882482 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.882463    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:48.882570 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:48.882540    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:18:48.883039 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.882895    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 12:18:48.883039 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.882959    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 12:18:48.883039 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.882959    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 12:18:48.883218 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.883032    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 12:18:48.883218 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.883117    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 12:18:48.884247 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.884103    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.884659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.884639    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zctxn\""
Jul 24 12:18:48.884770 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.884755    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 12:18:48.884942 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.884928    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 12:18:48.884942 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.884936    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ww6pg\""
Jul 24 12:18:48.885150 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.885059    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 12:18:48.885150 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.885129    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 12:18:48.885258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.885157    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 12:18:48.885357 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.885339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:48.885430 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:48.885409    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:18:48.886720 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.886704    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:18:48.886928 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.886909    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-99bkf\""
Jul 24 12:18:48.887051 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.887032    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:48.887295 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.887274    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 12:18:48.888361 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.888338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-44h9q"
Jul 24 12:18:48.889360 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.889342    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 12:18:48.890251 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.889797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:48.890251 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.889927    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 12:18:48.890251 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.890127    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 12:18:48.890251 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.890158    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ghsb5\""
Jul 24 12:18:48.891103 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.891082    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 12:18:48.891949 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.891930    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 12:18:48.892083 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.892065    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 12:18:48.892391 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.892371    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g528b\""
Jul 24 12:18:48.892391 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.892384    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:18:48.892503 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:48.892435    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:18:48.893275 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.893255    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 12:18:48.893532 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.893514    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 12:18:48.895723 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.895701    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bvjfd\""
Jul 24 12:18:48.899197 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3-tmp-dir\") pod \"node-resolver-hjxk2\" (UID: \"a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3\") " pod="openshift-dns/node-resolver-hjxk2"
Jul 24 12:18:48.899197 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-slash\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.899337 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-run-openvswitch\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.899337 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-run-k8s-cni-cncf-io\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.899337 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-multus-daemon-config\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.899337 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-host\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.899337 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899309    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:48.899337 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-log-socket\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.899604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899354    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.899604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-cni-bin\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.899604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-hostroot\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.899604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-run-multus-certs\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.899604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-registration-dir\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:48.899604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-kubelet\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.899604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899493    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32e6a712-727d-4045-927a-dd28b210caba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:48.899604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f905cc3c-2a1f-4cfc-934d-e465550ec974-serviceca\") pod \"node-ca-n6d7j\" (UID: \"f905cc3c-2a1f-4cfc-934d-e465550ec974\") " pod="openshift-image-registry/node-ca-n6d7j"
Jul 24 12:18:48.899604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-run-netns\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.899604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-cni-netd\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.899604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899583    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.899604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-kubernetes\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-sysctl-d\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-lib-modules\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6q78\" (UniqueName: \"kubernetes.io/projected/eb416fec-cb35-461e-8e44-7b1680a14b7e-kube-api-access-p6q78\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-kubelet-config\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899798    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgscv\" (UniqueName: \"kubernetes.io/projected/32e6a712-727d-4045-927a-dd28b210caba-kube-api-access-zgscv\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ca2b52-8079-468f-a135-55953a440641-host-slash\") pod \"iptables-alerter-bq9b6\" (UID: \"c1ca2b52-8079-468f-a135-55953a440641\") " pod="openshift-network-operator/iptables-alerter-bq9b6"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-var-lib-cni-bin\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3-hosts-file\") pod \"node-resolver-hjxk2\" (UID: \"a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3\") " pod="openshift-dns/node-resolver-hjxk2"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb416fec-cb35-461e-8e44-7b1680a14b7e-env-overrides\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899930    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb416fec-cb35-461e-8e44-7b1680a14b7e-ovn-node-metrics-cert\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32e6a712-727d-4045-927a-dd28b210caba-cnibin\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-cni-binary-copy\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.899993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-sysctl-conf\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900006    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-sys\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.900248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-socket-dir\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900068    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-device-dir\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-sys-fs\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdfh2\" (UniqueName: \"kubernetes.io/projected/7585202d-e3a1-46b8-90e8-316041c7e09f-kube-api-access-kdfh2\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900138    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-dbus\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900160    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32e6a712-727d-4045-927a-dd28b210caba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-var-lib-cni-multus\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-modprobe-d\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-run-systemd\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900250    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-run-ovn\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzl7x\" (UniqueName: \"kubernetes.io/projected/f905cc3c-2a1f-4cfc-934d-e465550ec974-kube-api-access-fzl7x\") pod \"node-ca-n6d7j\" (UID: \"f905cc3c-2a1f-4cfc-934d-e465550ec974\") " pod="openshift-image-registry/node-ca-n6d7j"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900295    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-multus-socket-dir-parent\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-systemd\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-var-lib-kubelet\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15f50eff-94fd-4dfe-8394-86cca2bfbd41-agent-certs\") pod \"konnectivity-agent-44h9q\" (UID: \"15f50eff-94fd-4dfe-8394-86cca2bfbd41\") " pod="kube-system/konnectivity-agent-44h9q"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-var-lib-openvswitch\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.900917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-node-log\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ca2b52-8079-468f-a135-55953a440641-iptables-alerter-script\") pod \"iptables-alerter-bq9b6\" (UID: \"c1ca2b52-8079-468f-a135-55953a440641\") " pod="openshift-network-operator/iptables-alerter-bq9b6"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxsf\" (UniqueName: \"kubernetes.io/projected/c1ca2b52-8079-468f-a135-55953a440641-kube-api-access-bnxsf\") pod \"iptables-alerter-bq9b6\" (UID: \"c1ca2b52-8079-468f-a135-55953a440641\") " pod="openshift-network-operator/iptables-alerter-bq9b6"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmwqc\" (UniqueName: \"kubernetes.io/projected/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-kube-api-access-jmwqc\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-os-release\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900664    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-sysconfig\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-tuned\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900708    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81ebb030-3e04-4442-a0fa-ce34c8ada49a-tmp\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f905cc3c-2a1f-4cfc-934d-e465550ec974-host\") pod \"node-ca-n6d7j\" (UID: \"f905cc3c-2a1f-4cfc-934d-e465550ec974\") " pod="openshift-image-registry/node-ca-n6d7j"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb416fec-cb35-461e-8e44-7b1680a14b7e-ovnkube-config\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-multus-cni-dir\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb416fec-cb35-461e-8e44-7b1680a14b7e-ovnkube-script-lib\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-var-lib-kubelet\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15f50eff-94fd-4dfe-8394-86cca2bfbd41-konnectivity-ca\") pod \"konnectivity-agent-44h9q\" (UID: \"15f50eff-94fd-4dfe-8394-86cca2bfbd41\") " pod="kube-system/konnectivity-agent-44h9q"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900888    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-systemd-units\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.901559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900912    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-etc-openvswitch\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32e6a712-727d-4045-927a-dd28b210caba-system-cni-dir\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32e6a712-727d-4045-927a-dd28b210caba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.900979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-system-cni-dir\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.901001    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-run-netns\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.901056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-multus-conf-dir\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.901094    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.901145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw66h\" (UniqueName: \"kubernetes.io/projected/477aa75c-bb3c-4355-8386-933be9dfe89b-kube-api-access-hw66h\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.901162    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-cnibin\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.901174    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkljv\" (UniqueName: \"kubernetes.io/projected/81ebb030-3e04-4442-a0fa-ce34c8ada49a-kube-api-access-zkljv\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.901187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qhs2\" (UniqueName: \"kubernetes.io/projected/a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3-kube-api-access-6qhs2\") pod \"node-resolver-hjxk2\" (UID: \"a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3\") " pod="openshift-dns/node-resolver-hjxk2"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.901211    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32e6a712-727d-4045-927a-dd28b210caba-os-release\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.901226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32e6a712-727d-4045-927a-dd28b210caba-cni-binary-copy\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.901267    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-etc-kubernetes\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:48.902181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.901287    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-run\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:48.903884 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.903862    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 12:18:48.928245 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.928224    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7zclm"
Jul 24 12:18:48.936995 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.936976    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7zclm"
Jul 24 12:18:48.979489 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.979458    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:18:48.989848 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:48.989829    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 12:18:49.001475 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15f50eff-94fd-4dfe-8394-86cca2bfbd41-agent-certs\") pod \"konnectivity-agent-44h9q\" (UID: \"15f50eff-94fd-4dfe-8394-86cca2bfbd41\") " pod="kube-system/konnectivity-agent-44h9q"
Jul 24 12:18:49.001475 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-var-lib-openvswitch\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.001652 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-node-log\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.001652 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:49.001652 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ca2b52-8079-468f-a135-55953a440641-iptables-alerter-script\") pod \"iptables-alerter-bq9b6\" (UID: \"c1ca2b52-8079-468f-a135-55953a440641\") " pod="openshift-network-operator/iptables-alerter-bq9b6"
Jul 24 12:18:49.001652 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxsf\" (UniqueName: \"kubernetes.io/projected/c1ca2b52-8079-468f-a135-55953a440641-kube-api-access-bnxsf\") pod \"iptables-alerter-bq9b6\" (UID: \"c1ca2b52-8079-468f-a135-55953a440641\") " pod="openshift-network-operator/iptables-alerter-bq9b6"
Jul 24 12:18:49.001652 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmwqc\" (UniqueName: \"kubernetes.io/projected/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-kube-api-access-jmwqc\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.001652 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-var-lib-openvswitch\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.001652 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.001654    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:18:49.001957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnt4\" (UniqueName: \"kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4\") pod \"network-check-target-xmjh2\" (UID: \"549ca555-24a4-4899-9071-f2776523a390\") " pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:18:49.001957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-node-log\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.001957 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.001750    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret podName:17a4fb33-440c-4563-b7e1-b9ee4ef34d90 nodeName:}" failed. No retries permitted until 2026-07-24 12:18:49.501702301 +0000 UTC m=+3.029449346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret") pod "global-pull-secret-syncer-gnrrg" (UID: "17a4fb33-440c-4563-b7e1-b9ee4ef34d90") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:18:49.001957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-os-release\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.001957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-sysconfig\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.001957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001863    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-tuned\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.001957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001893    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81ebb030-3e04-4442-a0fa-ce34c8ada49a-tmp\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.001957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001872    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 12:18:49.001957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001914    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-sysconfig\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.001957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f905cc3c-2a1f-4cfc-934d-e465550ec974-host\") pod \"node-ca-n6d7j\" (UID: \"f905cc3c-2a1f-4cfc-934d-e465550ec974\") " pod="openshift-image-registry/node-ca-n6d7j"
Jul 24 12:18:49.001957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001927    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-os-release\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001979    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f905cc3c-2a1f-4cfc-934d-e465550ec974-host\") pod \"node-ca-n6d7j\" (UID: \"f905cc3c-2a1f-4cfc-934d-e465550ec974\") " pod="openshift-image-registry/node-ca-n6d7j"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.001978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb416fec-cb35-461e-8e44-7b1680a14b7e-ovnkube-config\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002067    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-multus-cni-dir\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb416fec-cb35-461e-8e44-7b1680a14b7e-ovnkube-script-lib\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-var-lib-kubelet\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002154    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15f50eff-94fd-4dfe-8394-86cca2bfbd41-konnectivity-ca\") pod \"konnectivity-agent-44h9q\" (UID: \"15f50eff-94fd-4dfe-8394-86cca2bfbd41\") " pod="kube-system/konnectivity-agent-44h9q"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-systemd-units\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002199    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-etc-openvswitch\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002205    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-var-lib-kubelet\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002224    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32e6a712-727d-4045-927a-dd28b210caba-system-cni-dir\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32e6a712-727d-4045-927a-dd28b210caba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002277    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32e6a712-727d-4045-927a-dd28b210caba-system-cni-dir\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-system-cni-dir\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-run-netns\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-multus-conf-dir\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002360    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-system-cni-dir\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.002424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw66h\" (UniqueName: \"kubernetes.io/projected/477aa75c-bb3c-4355-8386-933be9dfe89b-kube-api-access-hw66h\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-cnibin\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkljv\" (UniqueName: \"kubernetes.io/projected/81ebb030-3e04-4442-a0fa-ce34c8ada49a-kube-api-access-zkljv\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qhs2\" (UniqueName: \"kubernetes.io/projected/a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3-kube-api-access-6qhs2\") pod \"node-resolver-hjxk2\" (UID: \"a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3\") " pod="openshift-dns/node-resolver-hjxk2"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32e6a712-727d-4045-927a-dd28b210caba-os-release\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32e6a712-727d-4045-927a-dd28b210caba-cni-binary-copy\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-etc-kubernetes\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-run\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002581    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3-tmp-dir\") pod \"node-resolver-hjxk2\" (UID: \"a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3\") " pod="openshift-dns/node-resolver-hjxk2"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002599    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb416fec-cb35-461e-8e44-7b1680a14b7e-ovnkube-config\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-slash\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002633    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-run-openvswitch\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-run-k8s-cni-cncf-io\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002711    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-systemd-units\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002768    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-etc-openvswitch\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15f50eff-94fd-4dfe-8394-86cca2bfbd41-konnectivity-ca\") pod \"konnectivity-agent-44h9q\" (UID: \"15f50eff-94fd-4dfe-8394-86cca2bfbd41\") " pod="kube-system/konnectivity-agent-44h9q"
Jul 24 12:18:49.002918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ca2b52-8079-468f-a135-55953a440641-iptables-alerter-script\") pod \"iptables-alerter-bq9b6\" (UID: \"c1ca2b52-8079-468f-a135-55953a440641\") " pod="openshift-network-operator/iptables-alerter-bq9b6"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002867    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-run\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32e6a712-727d-4045-927a-dd28b210caba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-run-netns\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.002314    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-multus-cni-dir\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32e6a712-727d-4045-927a-dd28b210caba-os-release\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-multus-conf-dir\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003070    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-etc-kubernetes\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-multus-daemon-config\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-host\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003217    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-cnibin\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-host\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003295    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-log-socket\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003343    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-run-openvswitch\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003355    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.003415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-cni-bin\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32e6a712-727d-4045-927a-dd28b210caba-cni-binary-copy\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003484    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb416fec-cb35-461e-8e44-7b1680a14b7e-ovnkube-script-lib\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003532    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-log-socket\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-cni-bin\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-hostroot\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003578    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-hostroot\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003582    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-slash\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-run-multus-certs\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-registration-dir\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003668    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-kubelet\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003690    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-run-multus-certs\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3-tmp-dir\") pod \"node-resolver-hjxk2\" (UID: \"a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3\") " pod="openshift-dns/node-resolver-hjxk2"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003708    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-registration-dir\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-kubelet\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32e6a712-727d-4045-927a-dd28b210caba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003760    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f905cc3c-2a1f-4cfc-934d-e465550ec974-serviceca\") pod \"node-ca-n6d7j\" (UID: \"f905cc3c-2a1f-4cfc-934d-e465550ec974\") " pod="openshift-image-registry/node-ca-n6d7j"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003784    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-run-netns\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-cni-netd\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003820    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-run-k8s-cni-cncf-io\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003834    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003859    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-multus-daemon-config\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-run-netns\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-kubernetes\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-cni-netd\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-sysctl-d\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-lib-modules\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6q78\" (UniqueName: \"kubernetes.io/projected/eb416fec-cb35-461e-8e44-7b1680a14b7e-kube-api-access-p6q78\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004032    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-sysctl-d\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-kubelet-config\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004050    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-lib-modules\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.004847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgscv\" (UniqueName: \"kubernetes.io/projected/32e6a712-727d-4045-927a-dd28b210caba-kube-api-access-zgscv\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-kubelet-config\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ca2b52-8079-468f-a135-55953a440641-host-slash\") pod \"iptables-alerter-bq9b6\" (UID: \"c1ca2b52-8079-468f-a135-55953a440641\") " pod="openshift-network-operator/iptables-alerter-bq9b6"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-kubernetes\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.004170    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f905cc3c-2a1f-4cfc-934d-e465550ec974-serviceca\") pod \"node-ca-n6d7j\" (UID: \"f905cc3c-2a1f-4cfc-934d-e465550ec974\") " pod="openshift-image-registry/node-ca-n6d7j"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.003939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004203    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32e6a712-727d-4045-927a-dd28b210caba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.004228    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs podName:7585202d-e3a1-46b8-90e8-316041c7e09f nodeName:}" failed. No retries permitted until 2026-07-24 12:18:49.504210921 +0000 UTC m=+3.031957974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs") pod "network-metrics-daemon-zdv26" (UID: "7585202d-e3a1-46b8-90e8-316041c7e09f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ca2b52-8079-468f-a135-55953a440641-host-slash\") pod \"iptables-alerter-bq9b6\" (UID: \"c1ca2b52-8079-468f-a135-55953a440641\") " pod="openshift-network-operator/iptables-alerter-bq9b6"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004249    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-var-lib-cni-bin\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-var-lib-cni-bin\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3-hosts-file\") pod \"node-resolver-hjxk2\" (UID: \"a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3\") " pod="openshift-dns/node-resolver-hjxk2"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb416fec-cb35-461e-8e44-7b1680a14b7e-env-overrides\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb416fec-cb35-461e-8e44-7b1680a14b7e-ovn-node-metrics-cert\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3-hosts-file\") pod \"node-resolver-hjxk2\" (UID: \"a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3\") " pod="openshift-dns/node-resolver-hjxk2"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32e6a712-727d-4045-927a-dd28b210caba-cnibin\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.005334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-cni-binary-copy\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004404    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-sysctl-conf\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-sys\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-socket-dir\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32e6a712-727d-4045-927a-dd28b210caba-cnibin\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-device-dir\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-sys\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004595    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-sysctl-conf\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004648    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-device-dir\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004742    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-socket-dir\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-sys-fs\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdfh2\" (UniqueName: \"kubernetes.io/projected/7585202d-e3a1-46b8-90e8-316041c7e09f-kube-api-access-kdfh2\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-dbus\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32e6a712-727d-4045-927a-dd28b210caba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-var-lib-cni-multus\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb416fec-cb35-461e-8e44-7b1680a14b7e-env-overrides\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004915    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-modprobe-d\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.005899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-run-systemd\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-run-ovn\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-cni-binary-copy\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.004987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzl7x\" (UniqueName: \"kubernetes.io/projected/f905cc3c-2a1f-4cfc-934d-e465550ec974-kube-api-access-fzl7x\") pod \"node-ca-n6d7j\" (UID: \"f905cc3c-2a1f-4cfc-934d-e465550ec974\") " pod="openshift-image-registry/node-ca-n6d7j"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-multus-socket-dir-parent\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-systemd\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-var-lib-kubelet\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-dbus\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-var-lib-kubelet\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/477aa75c-bb3c-4355-8386-933be9dfe89b-sys-fs\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005246    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32e6a712-727d-4045-927a-dd28b210caba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005252    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-modprobe-d\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005295    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-multus-socket-dir-parent\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005308    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-run-systemd\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-host-var-lib-cni-multus\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-systemd\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.006486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.005362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb416fec-cb35-461e-8e44-7b1680a14b7e-run-ovn\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.007247 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.007184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81ebb030-3e04-4442-a0fa-ce34c8ada49a-tmp\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.007311 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.007290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81ebb030-3e04-4442-a0fa-ce34c8ada49a-etc-tuned\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.007483 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.007461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15f50eff-94fd-4dfe-8394-86cca2bfbd41-agent-certs\") pod \"konnectivity-agent-44h9q\" (UID: \"15f50eff-94fd-4dfe-8394-86cca2bfbd41\") " pod="kube-system/konnectivity-agent-44h9q"
Jul 24 12:18:49.007643 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.007609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb416fec-cb35-461e-8e44-7b1680a14b7e-ovn-node-metrics-cert\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.012758 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.012735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qhs2\" (UniqueName: \"kubernetes.io/projected/a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3-kube-api-access-6qhs2\") pod \"node-resolver-hjxk2\" (UID: \"a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3\") " pod="openshift-dns/node-resolver-hjxk2"
Jul 24 12:18:49.012949 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.012934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxsf\" (UniqueName: \"kubernetes.io/projected/c1ca2b52-8079-468f-a135-55953a440641-kube-api-access-bnxsf\") pod \"iptables-alerter-bq9b6\" (UID: \"c1ca2b52-8079-468f-a135-55953a440641\") " pod="openshift-network-operator/iptables-alerter-bq9b6"
Jul 24 12:18:49.013156 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.013140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmwqc\" (UniqueName: \"kubernetes.io/projected/ffefdd54-35d0-4d95-a7fd-ded10dba56c9-kube-api-access-jmwqc\") pod \"multus-m4x75\" (UID: \"ffefdd54-35d0-4d95-a7fd-ded10dba56c9\") " pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.016344 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.016325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkljv\" (UniqueName: \"kubernetes.io/projected/81ebb030-3e04-4442-a0fa-ce34c8ada49a-kube-api-access-zkljv\") pod \"tuned-8j47g\" (UID: \"81ebb030-3e04-4442-a0fa-ce34c8ada49a\") " pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.016569 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.016529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdfh2\" (UniqueName: \"kubernetes.io/projected/7585202d-e3a1-46b8-90e8-316041c7e09f-kube-api-access-kdfh2\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:49.017145 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.017122    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6q78\" (UniqueName: \"kubernetes.io/projected/eb416fec-cb35-461e-8e44-7b1680a14b7e-kube-api-access-p6q78\") pod \"ovnkube-node-4ccs4\" (UID: \"eb416fec-cb35-461e-8e44-7b1680a14b7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.017638 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.017621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgscv\" (UniqueName: \"kubernetes.io/projected/32e6a712-727d-4045-927a-dd28b210caba-kube-api-access-zgscv\") pod \"multus-additional-cni-plugins-mm5x2\" (UID: \"32e6a712-727d-4045-927a-dd28b210caba\") " pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.018610 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.018594    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw66h\" (UniqueName: \"kubernetes.io/projected/477aa75c-bb3c-4355-8386-933be9dfe89b-kube-api-access-hw66h\") pod \"aws-ebs-csi-driver-node-6pptt\" (UID: \"477aa75c-bb3c-4355-8386-933be9dfe89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.018868 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.018848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzl7x\" (UniqueName: \"kubernetes.io/projected/f905cc3c-2a1f-4cfc-934d-e465550ec974-kube-api-access-fzl7x\") pod \"node-ca-n6d7j\" (UID: \"f905cc3c-2a1f-4cfc-934d-e465550ec974\") " pod="openshift-image-registry/node-ca-n6d7j"
Jul 24 12:18:49.106053 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.106001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnt4\" (UniqueName: \"kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4\") pod \"network-check-target-xmjh2\" (UID: \"549ca555-24a4-4899-9071-f2776523a390\") " pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:18:49.115005 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.114944    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:18:49.115005 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.114971    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:18:49.115005 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.114986    2566 projected.go:196] Error preparing data for projected volume kube-api-access-xgnt4 for pod openshift-network-diagnostics/network-check-target-xmjh2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:18:49.115214 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.115061    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4 podName:549ca555-24a4-4899-9071-f2776523a390 nodeName:}" failed. No retries permitted until 2026-07-24 12:18:49.615043818 +0000 UTC m=+3.142790866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xgnt4" (UniqueName: "kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4") pod "network-check-target-xmjh2" (UID: "549ca555-24a4-4899-9071-f2776523a390") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:18:49.188169 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.188136    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n6d7j"
Jul 24 12:18:49.197200 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.197175    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bq9b6"
Jul 24 12:18:49.207510 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.207477    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hjxk2"
Jul 24 12:18:49.212460 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.212437    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:18:49.219200 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.219184    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m4x75"
Jul 24 12:18:49.226974 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.226955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8j47g"
Jul 24 12:18:49.235650 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.235627    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt"
Jul 24 12:18:49.241662 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.241637    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-44h9q"
Jul 24 12:18:49.248261 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.248245    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mm5x2"
Jul 24 12:18:49.507840 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.507765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:49.507840 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.507810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:49.508060 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.507927    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:18:49.508060 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.507988    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs podName:7585202d-e3a1-46b8-90e8-316041c7e09f nodeName:}" failed. No retries permitted until 2026-07-24 12:18:50.507973645 +0000 UTC m=+4.035720676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs") pod "network-metrics-daemon-zdv26" (UID: "7585202d-e3a1-46b8-90e8-316041c7e09f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:18:49.508060 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.507927    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:18:49.508172 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.508069    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret podName:17a4fb33-440c-4563-b7e1-b9ee4ef34d90 nodeName:}" failed. No retries permitted until 2026-07-24 12:18:50.508057474 +0000 UTC m=+4.035804522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret") pod "global-pull-secret-syncer-gnrrg" (UID: "17a4fb33-440c-4563-b7e1-b9ee4ef34d90") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:18:49.673793 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:49.673766    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e6a712_727d_4045_927a_dd28b210caba.slice/crio-b0fc6f534712152eebdd854b23d3a80e35a2f40fe3e413758a08d3f9c5f1209c WatchSource:0}: Error finding container b0fc6f534712152eebdd854b23d3a80e35a2f40fe3e413758a08d3f9c5f1209c: Status 404 returned error can't find the container with id b0fc6f534712152eebdd854b23d3a80e35a2f40fe3e413758a08d3f9c5f1209c
Jul 24 12:18:49.676630 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:49.676603    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f50eff_94fd_4dfe_8394_86cca2bfbd41.slice/crio-f760b72882ce499882c73f1271ac280352d758ea2893e0adc0d470cf6c3933ee WatchSource:0}: Error finding container f760b72882ce499882c73f1271ac280352d758ea2893e0adc0d470cf6c3933ee: Status 404 returned error can't find the container with id f760b72882ce499882c73f1271ac280352d758ea2893e0adc0d470cf6c3933ee
Jul 24 12:18:49.679339 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:49.679318    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf905cc3c_2a1f_4cfc_934d_e465550ec974.slice/crio-661ac9af80fee17afa22d2d65d020737a0cc43d49deda2f981f5d3e8023f7435 WatchSource:0}: Error finding container 661ac9af80fee17afa22d2d65d020737a0cc43d49deda2f981f5d3e8023f7435: Status 404 returned error can't find the container with id 661ac9af80fee17afa22d2d65d020737a0cc43d49deda2f981f5d3e8023f7435
Jul 24 12:18:49.681125 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:49.681091    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ca2b52_8079_468f_a135_55953a440641.slice/crio-1896e156391bb5053aaa3c4a3ee39145b3f9377e26325fcf28b01662a5d951b6 WatchSource:0}: Error finding container 1896e156391bb5053aaa3c4a3ee39145b3f9377e26325fcf28b01662a5d951b6: Status 404 returned error can't find the container with id 1896e156391bb5053aaa3c4a3ee39145b3f9377e26325fcf28b01662a5d951b6
Jul 24 12:18:49.682430 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:49.682405    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb416fec_cb35_461e_8e44_7b1680a14b7e.slice/crio-ca93c5dfeb493f3ec7e4931641b94ca7064362690698b9f4bbd3bf4f7d5afed4 WatchSource:0}: Error finding container ca93c5dfeb493f3ec7e4931641b94ca7064362690698b9f4bbd3bf4f7d5afed4: Status 404 returned error can't find the container with id ca93c5dfeb493f3ec7e4931641b94ca7064362690698b9f4bbd3bf4f7d5afed4
Jul 24 12:18:49.683277 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:49.683256    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477aa75c_bb3c_4355_8386_933be9dfe89b.slice/crio-16b66d48139ed8b2a8b75e08d0b0d6113b3cebf203235a9c6e14393fd0e9b983 WatchSource:0}: Error finding container 16b66d48139ed8b2a8b75e08d0b0d6113b3cebf203235a9c6e14393fd0e9b983: Status 404 returned error can't find the container with id 16b66d48139ed8b2a8b75e08d0b0d6113b3cebf203235a9c6e14393fd0e9b983
Jul 24 12:18:49.685662 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:49.685635    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffefdd54_35d0_4d95_a7fd_ded10dba56c9.slice/crio-82b4b971633ac7c94b5c3a63fe9917f3392e4870fd3d65d43373abccaf5501bd WatchSource:0}: Error finding container 82b4b971633ac7c94b5c3a63fe9917f3392e4870fd3d65d43373abccaf5501bd: Status 404 returned error can't find the container with id 82b4b971633ac7c94b5c3a63fe9917f3392e4870fd3d65d43373abccaf5501bd
Jul 24 12:18:49.686422 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:49.686386    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ebb030_3e04_4442_a0fa_ce34c8ada49a.slice/crio-d546926a240773a6095dd11ba2ac753cbe098af02755161a8edd06e98be02439 WatchSource:0}: Error finding container d546926a240773a6095dd11ba2ac753cbe098af02755161a8edd06e98be02439: Status 404 returned error can't find the container with id d546926a240773a6095dd11ba2ac753cbe098af02755161a8edd06e98be02439
Jul 24 12:18:49.692968 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:18:49.692951    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20104cb88cafd62fcd060f78a1481bf.slice/crio-da80548babfcf1857e1aa3a1067e191809816134fad6cb9725983527a74b7497 WatchSource:0}: Error finding container da80548babfcf1857e1aa3a1067e191809816134fad6cb9725983527a74b7497: Status 404 returned error can't find the container with id da80548babfcf1857e1aa3a1067e191809816134fad6cb9725983527a74b7497
Jul 24 12:18:49.708951 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.708932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnt4\" (UniqueName: \"kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4\") pod \"network-check-target-xmjh2\" (UID: \"549ca555-24a4-4899-9071-f2776523a390\") " pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:18:49.709059 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.709048    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:18:49.709102 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.709064    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:18:49.709102 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.709082    2566 projected.go:196] Error preparing data for projected volume kube-api-access-xgnt4 for pod openshift-network-diagnostics/network-check-target-xmjh2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:18:49.709165 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.709125    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4 podName:549ca555-24a4-4899-9071-f2776523a390 nodeName:}" failed. No retries permitted until 2026-07-24 12:18:50.709111514 +0000 UTC m=+4.236858545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xgnt4" (UniqueName: "kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4") pod "network-check-target-xmjh2" (UID: "549ca555-24a4-4899-9071-f2776523a390") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:18:49.940345 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.940308    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 12:13:48 +0000 UTC" deadline="2028-03-19 06:37:41.951763218 +0000 UTC"
Jul 24 12:18:49.940345 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.940346    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14490h18m52.011425958s"
Jul 24 12:18:49.969658 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.969632    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:49.969751 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:49.969734    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:18:49.976778 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.976751    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal" event={"ID":"f20104cb88cafd62fcd060f78a1481bf","Type":"ContainerStarted","Data":"da80548babfcf1857e1aa3a1067e191809816134fad6cb9725983527a74b7497"}
Jul 24 12:18:49.977845 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.977823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt" event={"ID":"477aa75c-bb3c-4355-8386-933be9dfe89b","Type":"ContainerStarted","Data":"16b66d48139ed8b2a8b75e08d0b0d6113b3cebf203235a9c6e14393fd0e9b983"}
Jul 24 12:18:49.978968 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.978950    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4" event={"ID":"eb416fec-cb35-461e-8e44-7b1680a14b7e","Type":"ContainerStarted","Data":"ca93c5dfeb493f3ec7e4931641b94ca7064362690698b9f4bbd3bf4f7d5afed4"}
Jul 24 12:18:49.979959 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.979937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n6d7j" event={"ID":"f905cc3c-2a1f-4cfc-934d-e465550ec974","Type":"ContainerStarted","Data":"661ac9af80fee17afa22d2d65d020737a0cc43d49deda2f981f5d3e8023f7435"}
Jul 24 12:18:49.980938 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.980916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-44h9q" event={"ID":"15f50eff-94fd-4dfe-8394-86cca2bfbd41","Type":"ContainerStarted","Data":"f760b72882ce499882c73f1271ac280352d758ea2893e0adc0d470cf6c3933ee"}
Jul 24 12:18:49.981992 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.981975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm5x2" event={"ID":"32e6a712-727d-4045-927a-dd28b210caba","Type":"ContainerStarted","Data":"b0fc6f534712152eebdd854b23d3a80e35a2f40fe3e413758a08d3f9c5f1209c"}
Jul 24 12:18:49.982966 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.982941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hjxk2" event={"ID":"a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3","Type":"ContainerStarted","Data":"5920b50c180a1b9a5d3b9036410888cc634f4620be4485fdc44a5a4933b32bde"}
Jul 24 12:18:49.984652 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.984630    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-61.ec2.internal" event={"ID":"017919978be3c1d1d7d67e195198972d","Type":"ContainerStarted","Data":"17fbd07c03a6ddcfd6893607455457a11deae90e5286b30be0b4e3a8fc2bbf95"}
Jul 24 12:18:49.985627 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.985610    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8j47g" event={"ID":"81ebb030-3e04-4442-a0fa-ce34c8ada49a","Type":"ContainerStarted","Data":"d546926a240773a6095dd11ba2ac753cbe098af02755161a8edd06e98be02439"}
Jul 24 12:18:49.986600 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.986583    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m4x75" event={"ID":"ffefdd54-35d0-4d95-a7fd-ded10dba56c9","Type":"ContainerStarted","Data":"82b4b971633ac7c94b5c3a63fe9917f3392e4870fd3d65d43373abccaf5501bd"}
Jul 24 12:18:49.987534 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:49.987518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bq9b6" event={"ID":"c1ca2b52-8079-468f-a135-55953a440641","Type":"ContainerStarted","Data":"1896e156391bb5053aaa3c4a3ee39145b3f9377e26325fcf28b01662a5d951b6"}
Jul 24 12:18:50.003518 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:50.003479    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-61.ec2.internal" podStartSLOduration=2.003465478 podStartE2EDuration="2.003465478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:18:50.003048326 +0000 UTC m=+3.530795376" watchObservedRunningTime="2026-07-24 12:18:50.003465478 +0000 UTC m=+3.531212534"
Jul 24 12:18:50.515212 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:50.515142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:50.515342 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:50.515229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:50.515415 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:50.515345    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:18:50.515415 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:50.515404    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs podName:7585202d-e3a1-46b8-90e8-316041c7e09f nodeName:}" failed. No retries permitted until 2026-07-24 12:18:52.51538623 +0000 UTC m=+6.043133264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs") pod "network-metrics-daemon-zdv26" (UID: "7585202d-e3a1-46b8-90e8-316041c7e09f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:18:50.515838 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:50.515816    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:18:50.515937 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:50.515870    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret podName:17a4fb33-440c-4563-b7e1-b9ee4ef34d90 nodeName:}" failed. No retries permitted until 2026-07-24 12:18:52.515855878 +0000 UTC m=+6.043602912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret") pod "global-pull-secret-syncer-gnrrg" (UID: "17a4fb33-440c-4563-b7e1-b9ee4ef34d90") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:18:50.716575 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:50.716475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnt4\" (UniqueName: \"kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4\") pod \"network-check-target-xmjh2\" (UID: \"549ca555-24a4-4899-9071-f2776523a390\") " pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:18:50.716761 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:50.716680    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:18:50.716761 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:50.716701    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:18:50.716761 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:50.716713    2566 projected.go:196] Error preparing data for projected volume kube-api-access-xgnt4 for pod openshift-network-diagnostics/network-check-target-xmjh2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:18:50.716914 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:50.716770    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4 podName:549ca555-24a4-4899-9071-f2776523a390 nodeName:}" failed. No retries permitted until 2026-07-24 12:18:52.716752409 +0000 UTC m=+6.244499454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xgnt4" (UniqueName: "kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4") pod "network-check-target-xmjh2" (UID: "549ca555-24a4-4899-9071-f2776523a390") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:18:50.940974 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:50.940687    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 12:13:48 +0000 UTC" deadline="2028-01-09 14:31:57.552623949 +0000 UTC"
Jul 24 12:18:50.940974 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:50.940975    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12818h13m6.611656323s"
Jul 24 12:18:50.972707 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:50.972676    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:18:50.972872 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:50.972807    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:18:50.972940 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:50.972891    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:50.972990 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:50.972974    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:18:51.969100 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:51.969065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:51.969498 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:51.969205    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:18:52.531968 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:52.531931    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:52.532154 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:52.532038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:52.532227 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:52.532173    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:18:52.532281 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:52.532235    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs podName:7585202d-e3a1-46b8-90e8-316041c7e09f nodeName:}" failed. No retries permitted until 2026-07-24 12:18:56.53221546 +0000 UTC m=+10.059962494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs") pod "network-metrics-daemon-zdv26" (UID: "7585202d-e3a1-46b8-90e8-316041c7e09f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:18:52.532650 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:52.532627    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:18:52.532717 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:52.532688    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret podName:17a4fb33-440c-4563-b7e1-b9ee4ef34d90 nodeName:}" failed. No retries permitted until 2026-07-24 12:18:56.532672465 +0000 UTC m=+10.060419520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret") pod "global-pull-secret-syncer-gnrrg" (UID: "17a4fb33-440c-4563-b7e1-b9ee4ef34d90") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:18:52.734384 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:52.734307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnt4\" (UniqueName: \"kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4\") pod \"network-check-target-xmjh2\" (UID: \"549ca555-24a4-4899-9071-f2776523a390\") " pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:18:52.734558 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:52.734484    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:18:52.734558 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:52.734505    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:18:52.734558 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:52.734518    2566 projected.go:196] Error preparing data for projected volume kube-api-access-xgnt4 for pod openshift-network-diagnostics/network-check-target-xmjh2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:18:52.734719 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:52.734575    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4 podName:549ca555-24a4-4899-9071-f2776523a390 nodeName:}" failed. No retries permitted until 2026-07-24 12:18:56.734557594 +0000 UTC m=+10.262304629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xgnt4" (UniqueName: "kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4") pod "network-check-target-xmjh2" (UID: "549ca555-24a4-4899-9071-f2776523a390") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:18:52.970195 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:52.970117    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:52.970620 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:52.970295    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:18:52.970725 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:52.970682    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:18:52.970821 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:52.970770    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:18:53.969210 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:53.969176    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:53.969378 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:53.969307    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:18:54.971161 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:54.971131    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:54.971628 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:54.971136    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:18:54.971628 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:54.971270    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:18:54.971628 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:54.971343    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:18:55.969348 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:55.968898    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:55.969348 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:55.969002    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:18:56.569445 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:56.569189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:56.569445 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:56.569261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:56.569445 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:56.569315    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:18:56.569445 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:56.569367    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:18:56.569445 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:56.569377    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs podName:7585202d-e3a1-46b8-90e8-316041c7e09f nodeName:}" failed. No retries permitted until 2026-07-24 12:19:04.569362359 +0000 UTC m=+18.097109394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs") pod "network-metrics-daemon-zdv26" (UID: "7585202d-e3a1-46b8-90e8-316041c7e09f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:18:56.569445 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:56.569418    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret podName:17a4fb33-440c-4563-b7e1-b9ee4ef34d90 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:04.569401767 +0000 UTC m=+18.097148804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret") pod "global-pull-secret-syncer-gnrrg" (UID: "17a4fb33-440c-4563-b7e1-b9ee4ef34d90") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:18:56.770999 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:56.770960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnt4\" (UniqueName: \"kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4\") pod \"network-check-target-xmjh2\" (UID: \"549ca555-24a4-4899-9071-f2776523a390\") " pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:18:56.771193 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:56.771133    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:18:56.771193 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:56.771149    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:18:56.771193 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:56.771157    2566 projected.go:196] Error preparing data for projected volume kube-api-access-xgnt4 for pod openshift-network-diagnostics/network-check-target-xmjh2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:18:56.771362 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:56.771205    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4 podName:549ca555-24a4-4899-9071-f2776523a390 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:04.771189627 +0000 UTC m=+18.298936674 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xgnt4" (UniqueName: "kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4") pod "network-check-target-xmjh2" (UID: "549ca555-24a4-4899-9071-f2776523a390") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:18:56.969866 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:56.969801    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:18:56.970012 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:56.969882    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:18:56.970012 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:56.969934    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:56.970012 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:56.969982    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:18:57.970011 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:57.969976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:57.970476 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:57.970129    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:18:58.970715 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:58.970128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:18:58.970715 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:58.970154    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:18:58.970715 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:58.970271    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:18:58.970715 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:58.970672    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:18:59.969118 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:18:59.969088    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:18:59.969315 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:18:59.969215    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:19:00.969305 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:00.969274    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:00.969674 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:00.969393    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:19:00.969674 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:00.969445    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:00.969674 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:00.969536    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:19:01.968865 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:01.968833    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:01.969052 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:01.968959    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:19:02.968940 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:02.968899    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:02.969421 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:02.968900    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:02.969421 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:02.969150    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:19:02.969421 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:02.969046    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:19:03.968702 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:03.968651    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:03.968906 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:03.968798    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:19:04.630830 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:04.630797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:04.631405 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:04.630847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:04.631405 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:04.630952    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:04.631405 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:04.630963    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:04.631405 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:04.631030    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret podName:17a4fb33-440c-4563-b7e1-b9ee4ef34d90 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:20.63099526 +0000 UTC m=+34.158742294 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret") pod "global-pull-secret-syncer-gnrrg" (UID: "17a4fb33-440c-4563-b7e1-b9ee4ef34d90") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:04.631405 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:04.631050    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs podName:7585202d-e3a1-46b8-90e8-316041c7e09f nodeName:}" failed. No retries permitted until 2026-07-24 12:19:20.631039217 +0000 UTC m=+34.158786252 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs") pod "network-metrics-daemon-zdv26" (UID: "7585202d-e3a1-46b8-90e8-316041c7e09f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:04.832430 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:04.832398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnt4\" (UniqueName: \"kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4\") pod \"network-check-target-xmjh2\" (UID: \"549ca555-24a4-4899-9071-f2776523a390\") " pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:04.832592 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:04.832576    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:19:04.832633 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:04.832602    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:19:04.832633 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:04.832613    2566 projected.go:196] Error preparing data for projected volume kube-api-access-xgnt4 for pod openshift-network-diagnostics/network-check-target-xmjh2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:04.832693 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:04.832665    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4 podName:549ca555-24a4-4899-9071-f2776523a390 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:20.83264975 +0000 UTC m=+34.360396798 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xgnt4" (UniqueName: "kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4") pod "network-check-target-xmjh2" (UID: "549ca555-24a4-4899-9071-f2776523a390") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:04.970525 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:04.969926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:04.970525 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:04.970110    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:19:04.972681 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:04.971112    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:04.972681 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:04.971283    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:19:05.968797 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:05.968763    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:05.969289 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:05.968887    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:19:06.969568 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:06.969533    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:06.970062 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:06.969655    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:19:06.970062 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:06.969718    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:06.970062 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:06.969777    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:19:07.969219 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:07.969186    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:07.969405 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:07.969292    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:19:08.969289 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:08.969258    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:08.969655 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:08.969388    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:19:08.969655 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:08.969632    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:08.969765 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:08.969742    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:19:09.048401 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:09.048367    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8j47g" event={"ID":"81ebb030-3e04-4442-a0fa-ce34c8ada49a","Type":"ContainerStarted","Data":"e8a3abbe61d6eeb9688aef3b0632ea5e319d1be566064ef6f296cfeaab6fa84f"}
Jul 24 12:19:09.052577 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:09.052551    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m4x75" event={"ID":"ffefdd54-35d0-4d95-a7fd-ded10dba56c9","Type":"ContainerStarted","Data":"90cb0b960eebb8bae178db62703f485575192a7622e1989ea6de2cc0eca5c028"}
Jul 24 12:19:09.056667 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:09.056581    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4" event={"ID":"eb416fec-cb35-461e-8e44-7b1680a14b7e","Type":"ContainerStarted","Data":"245c1844f6f9ebb94c5c15dd57493844d2cbb3482c372be8b5e8f0040e58fdd4"}
Jul 24 12:19:09.125307 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:09.121614    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8j47g" podStartSLOduration=2.963173649 podStartE2EDuration="22.121570042s" totalImagesPullingTime="19.158396393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:09.081936656 +0000 UTC m=+22.609683723" watchObservedRunningTime="2026-07-24 12:19:09.121570042 +0000 UTC m=+22.649317097"
Jul 24 12:19:09.125307 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:09.122811    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-m4x75" podStartSLOduration=2.9226691000000002 podStartE2EDuration="22.122800918s" totalImagesPullingTime="19.200131818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:09.121348895 +0000 UTC m=+22.649095950" watchObservedRunningTime="2026-07-24 12:19:09.122800918 +0000 UTC m=+22.650547974"
Jul 24 12:19:09.968731 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:09.968702    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:09.968907 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:09.968806    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:19:10.060743 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.060705    2566 generic.go:356] "Generic (PLEG): container finished" podID="f20104cb88cafd62fcd060f78a1481bf" containerID="9e9f95d5ee6bf3ea0288dde7d5138acd0f91094f0f6bf4323785d70e35068f51" exitCode=0
Jul 24 12:19:10.061643 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.060801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal" event={"ID":"f20104cb88cafd62fcd060f78a1481bf","Type":"ContainerDied","Data":"9e9f95d5ee6bf3ea0288dde7d5138acd0f91094f0f6bf4323785d70e35068f51"}
Jul 24 12:19:10.062806 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.062640    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt" event={"ID":"477aa75c-bb3c-4355-8386-933be9dfe89b","Type":"ContainerStarted","Data":"87c7c5a459767b83fdf8fec13d09774bce892984b994107a3ce055c9af7f2acb"}
Jul 24 12:19:10.066412 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.066385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4" event={"ID":"eb416fec-cb35-461e-8e44-7b1680a14b7e","Type":"ContainerStarted","Data":"9aad2cba928601bfbdf11ab0273934698415ea92643aaa21337c2024f4806ab1"}
Jul 24 12:19:10.066523 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.066464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4" event={"ID":"eb416fec-cb35-461e-8e44-7b1680a14b7e","Type":"ContainerStarted","Data":"9d986e36c8f009ec229cabd6b433080224a1da229e7f03fe4cd392520925e5e7"}
Jul 24 12:19:10.066523 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.066514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4" event={"ID":"eb416fec-cb35-461e-8e44-7b1680a14b7e","Type":"ContainerStarted","Data":"ce5fb1605efbfdb9869b94e666fb0f0120a81792374bfe12b2206fd87c64013a"}
Jul 24 12:19:10.066631 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.066560    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4" event={"ID":"eb416fec-cb35-461e-8e44-7b1680a14b7e","Type":"ContainerStarted","Data":"711c91b74dc4050f664f53bf3cbca9b711d9c46d844017dbad12bc33f88bbbbf"}
Jul 24 12:19:10.066631 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.066609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4" event={"ID":"eb416fec-cb35-461e-8e44-7b1680a14b7e","Type":"ContainerStarted","Data":"548d75ec4d138b618852e910d453f34ebc5a665ceea2031e455f2f11d95469b0"}
Jul 24 12:19:10.068121 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.068086    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n6d7j" event={"ID":"f905cc3c-2a1f-4cfc-934d-e465550ec974","Type":"ContainerStarted","Data":"b1da2a23a389d4a16533b1faa45e7c3cabacf4864234551a6f1b3b83823e8e34"}
Jul 24 12:19:10.069874 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.069537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-44h9q" event={"ID":"15f50eff-94fd-4dfe-8394-86cca2bfbd41","Type":"ContainerStarted","Data":"ac70dd8f7cf437d7e3090210c1ebcb26e9b427c1e7edc4de1200baa9be1cb1a4"}
Jul 24 12:19:10.071247 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.071218    2566 generic.go:356] "Generic (PLEG): container finished" podID="32e6a712-727d-4045-927a-dd28b210caba" containerID="dc743f4a16d350c4d68dfc565a97d99b9395b1c6f196a37459d8b485346c2493" exitCode=0
Jul 24 12:19:10.071342 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.071301    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm5x2" event={"ID":"32e6a712-727d-4045-927a-dd28b210caba","Type":"ContainerDied","Data":"dc743f4a16d350c4d68dfc565a97d99b9395b1c6f196a37459d8b485346c2493"}
Jul 24 12:19:10.073512 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.073483    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hjxk2" event={"ID":"a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3","Type":"ContainerStarted","Data":"b0f4987ce8d7a5152bc6b19394a6b1d21a79ecdd9d949e4d4d27a893ccfac404"}
Jul 24 12:19:10.123790 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.123683    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hjxk2" podStartSLOduration=3.949614694 podStartE2EDuration="23.123667613s" totalImagesPullingTime="19.174052919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:10.12288021 +0000 UTC m=+23.650627263" watchObservedRunningTime="2026-07-24 12:19:10.123667613 +0000 UTC m=+23.651414666"
Jul 24 12:19:10.139780 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.139742    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-44h9q" podStartSLOduration=3.988471887 podStartE2EDuration="23.139728877s" totalImagesPullingTime="19.15125699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:10.139571696 +0000 UTC m=+23.667318728" watchObservedRunningTime="2026-07-24 12:19:10.139728877 +0000 UTC m=+23.667475930"
Jul 24 12:19:10.156815 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.156776    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n6d7j" podStartSLOduration=7.957906345 podStartE2EDuration="23.15676322s" totalImagesPullingTime="15.198856875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:10.156751167 +0000 UTC m=+23.684498220" watchObservedRunningTime="2026-07-24 12:19:10.15676322 +0000 UTC m=+23.684510273"
Jul 24 12:19:10.644352 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.644325    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 12:19:10.969631 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.969485    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T12:19:10.644346707Z","UUID":"b7e2deb2-3f62-4dfa-a8c8-71f8d9fd5d60","Handler":null,"Name":"","Endpoint":""}
Jul 24 12:19:10.969863 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.969651    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:10.969863 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.969686    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:10.969863 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:10.969772    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:19:10.970044 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:10.969899    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:19:10.971071 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.971054    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 12:19:10.971166 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:10.971086    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 12:19:11.077533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:11.077497    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt" event={"ID":"477aa75c-bb3c-4355-8386-933be9dfe89b","Type":"ContainerStarted","Data":"93bdbfa3b2925067f1b981262e8ad725cadd7f170652ad00bb965b0477178c93"}
Jul 24 12:19:11.078863 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:11.078834    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bq9b6" event={"ID":"c1ca2b52-8079-468f-a135-55953a440641","Type":"ContainerStarted","Data":"e6e34636e7af4c3887b1542f7a6bad2779983adfed6ae14eec6a6c1a0c65522f"}
Jul 24 12:19:11.080680 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:11.080650    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal" event={"ID":"f20104cb88cafd62fcd060f78a1481bf","Type":"ContainerStarted","Data":"64525f12fefaf90e3981ef738da2eebb13e4447000b9390f048723d7e36e0af2"}
Jul 24 12:19:11.094826 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:11.094786    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bq9b6" podStartSLOduration=4.937787501 podStartE2EDuration="24.094775835s" totalImagesPullingTime="19.156988334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:11.094669789 +0000 UTC m=+24.622416842" watchObservedRunningTime="2026-07-24 12:19:11.094775835 +0000 UTC m=+24.622522888"
Jul 24 12:19:11.110350 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:11.110314    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-61.ec2.internal" podStartSLOduration=23.110303363 podStartE2EDuration="23.110303363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:11.109963189 +0000 UTC m=+24.637710242" watchObservedRunningTime="2026-07-24 12:19:11.110303363 +0000 UTC m=+24.638050417"
Jul 24 12:19:11.969071 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:11.969047    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:11.969237 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:11.969167    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:19:12.086858 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:12.086823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4" event={"ID":"eb416fec-cb35-461e-8e44-7b1680a14b7e","Type":"ContainerStarted","Data":"521b6935c2da59b341e70765fe4c5a1cc70a3f4ee2b0091075f18b75303af8d2"}
Jul 24 12:19:12.457180 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:12.457144    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-44h9q"
Jul 24 12:19:12.457841 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:12.457815    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-44h9q"
Jul 24 12:19:12.969156 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:12.969125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:12.969351 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:12.969125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:12.969351 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:12.969253    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:19:12.969478 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:12.969364    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:19:13.091382 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:13.091346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt" event={"ID":"477aa75c-bb3c-4355-8386-933be9dfe89b","Type":"ContainerStarted","Data":"ed3f6d7d0f35b6869ddd7e1c4767925adf93bbb41942da45f6b4c17cbc8c07d1"}
Jul 24 12:19:13.091899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:13.091825    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-44h9q"
Jul 24 12:19:13.092315 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:13.092297    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-44h9q"
Jul 24 12:19:13.121826 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:13.121779    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pptt" podStartSLOduration=4.37856987 podStartE2EDuration="26.121763303s" totalImagesPullingTime="21.743193433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:13.121449587 +0000 UTC m=+26.649196644" watchObservedRunningTime="2026-07-24 12:19:13.121763303 +0000 UTC m=+26.649510356"
Jul 24 12:19:13.969576 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:13.969544    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:13.969771 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:13.969549    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:13.969771 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:13.969706    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:19:13.969892 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:13.969838    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:19:14.969357 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:14.969210    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:14.969816 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:14.969443    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:19:15.099491 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:15.099453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4" event={"ID":"eb416fec-cb35-461e-8e44-7b1680a14b7e","Type":"ContainerStarted","Data":"2cd19c7f86ffff63764101dcf4a3d42f98c4f9b2c249ddb8dbfff71fc39bc2e2"}
Jul 24 12:19:15.099928 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:15.099734    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:19:15.101503 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:15.101479    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm5x2" event={"ID":"32e6a712-727d-4045-927a-dd28b210caba","Type":"ContainerStarted","Data":"e6f319937e19835dd12fea63a767bb6c54a6f520169f1f192611e5cfef276ffe"}
Jul 24 12:19:15.119897 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:15.119876    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:19:15.131455 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:15.131389    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4" podStartSLOduration=8.903381696 podStartE2EDuration="28.131373795s" totalImagesPullingTime="19.227992099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:15.130855145 +0000 UTC m=+28.658602200" watchObservedRunningTime="2026-07-24 12:19:15.131373795 +0000 UTC m=+28.659120849"
Jul 24 12:19:15.969476 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:15.969447    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:15.970174 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:15.969447    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:15.970174 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:15.969554    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:19:15.970174 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:15.969615    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:19:16.105097 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:16.105066    2566 generic.go:356] "Generic (PLEG): container finished" podID="32e6a712-727d-4045-927a-dd28b210caba" containerID="e6f319937e19835dd12fea63a767bb6c54a6f520169f1f192611e5cfef276ffe" exitCode=0
Jul 24 12:19:16.105273 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:16.105153    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm5x2" event={"ID":"32e6a712-727d-4045-927a-dd28b210caba","Type":"ContainerDied","Data":"e6f319937e19835dd12fea63a767bb6c54a6f520169f1f192611e5cfef276ffe"}
Jul 24 12:19:16.105582 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:16.105560    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:19:16.105657 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:16.105636    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:19:16.121430 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:16.121405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:19:16.972772 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:16.972744    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:16.973232 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:16.972868    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:19:16.994244 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:16.993653    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xmjh2"]
Jul 24 12:19:16.994244 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:16.993775    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:16.994244 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:16.993872    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:19:16.994619 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:16.994597    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zdv26"]
Jul 24 12:19:16.995440 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:16.995407    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gnrrg"]
Jul 24 12:19:16.995548 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:16.995501    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:16.995627 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:16.995605    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:19:17.107945 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:17.107700    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:17.108332 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:17.108277    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:19:18.111637 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:18.111604    2566 generic.go:356] "Generic (PLEG): container finished" podID="32e6a712-727d-4045-927a-dd28b210caba" containerID="94504a2310bb57bd6c214243056162c8c9b4c40c5fe6ce0026d2d90fcfeaaf84" exitCode=0
Jul 24 12:19:18.112011 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:18.111700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm5x2" event={"ID":"32e6a712-727d-4045-927a-dd28b210caba","Type":"ContainerDied","Data":"94504a2310bb57bd6c214243056162c8c9b4c40c5fe6ce0026d2d90fcfeaaf84"}
Jul 24 12:19:18.969634 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:18.969540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:18.969634 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:18.969538    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:18.969841 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:18.969666    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:18.969841 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:18.969683    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:19:18.969841 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:18.969752    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:19:18.970299 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:18.969864    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:19:20.117750 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:20.117720    2566 generic.go:356] "Generic (PLEG): container finished" podID="32e6a712-727d-4045-927a-dd28b210caba" containerID="f9291f729bad9cdac776d74e766932ffbaf897e5d0a8d3b2aaa9001cd1149f49" exitCode=0
Jul 24 12:19:20.118153 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:20.117762    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm5x2" event={"ID":"32e6a712-727d-4045-927a-dd28b210caba","Type":"ContainerDied","Data":"f9291f729bad9cdac776d74e766932ffbaf897e5d0a8d3b2aaa9001cd1149f49"}
Jul 24 12:19:20.646216 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:20.646178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:20.646383 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:20.646241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:20.646383 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:20.646316    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:20.646383 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:20.646378    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret podName:17a4fb33-440c-4563-b7e1-b9ee4ef34d90 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:52.646362397 +0000 UTC m=+66.174109431 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret") pod "global-pull-secret-syncer-gnrrg" (UID: "17a4fb33-440c-4563-b7e1-b9ee4ef34d90") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:20.646493 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:20.646319    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:20.646493 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:20.646455    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs podName:7585202d-e3a1-46b8-90e8-316041c7e09f nodeName:}" failed. No retries permitted until 2026-07-24 12:19:52.646441731 +0000 UTC m=+66.174188763 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs") pod "network-metrics-daemon-zdv26" (UID: "7585202d-e3a1-46b8-90e8-316041c7e09f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:20.847041 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:20.846991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnt4\" (UniqueName: \"kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4\") pod \"network-check-target-xmjh2\" (UID: \"549ca555-24a4-4899-9071-f2776523a390\") " pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:20.847196 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:20.847151    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:19:20.847196 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:20.847169    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:19:20.847196 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:20.847178    2566 projected.go:196] Error preparing data for projected volume kube-api-access-xgnt4 for pod openshift-network-diagnostics/network-check-target-xmjh2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:20.847304 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:20.847227    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4 podName:549ca555-24a4-4899-9071-f2776523a390 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:52.847213784 +0000 UTC m=+66.374960816 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xgnt4" (UniqueName: "kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4") pod "network-check-target-xmjh2" (UID: "549ca555-24a4-4899-9071-f2776523a390") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:20.969515 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:20.969416    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:20.969515 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:20.969416    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:20.969722 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:20.969416    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:20.969722 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:20.969549    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gnrrg" podUID="17a4fb33-440c-4563-b7e1-b9ee4ef34d90"
Jul 24 12:19:20.969722 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:20.969633    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmjh2" podUID="549ca555-24a4-4899-9071-f2776523a390"
Jul 24 12:19:20.969899 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:20.969733    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zdv26" podUID="7585202d-e3a1-46b8-90e8-316041c7e09f"
Jul 24 12:19:21.779728 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.779702    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-61.ec2.internal" event="NodeReady"
Jul 24 12:19:21.780170 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.779838    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 12:19:21.819253 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.819227    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-847cc74b65-87hbw"]
Jul 24 12:19:21.822551 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.822521    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:21.822681 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.822565    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"]
Jul 24 12:19:21.825462 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.825438    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"]
Jul 24 12:19:21.825637 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.825618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:21.829357 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.829328    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xlzt9\""
Jul 24 12:19:21.829836 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.829795    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 12:19:21.830852 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.830822    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl"]
Jul 24 12:19:21.831199 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.831168    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 12:19:21.831527 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.831482    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 12:19:21.831948 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.831908    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:21.832055 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.831971    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 12:19:21.834422 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.834400    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 24 12:19:21.834694 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.834674    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 12:19:21.838065 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.835578    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl"
Jul 24 12:19:21.838065 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.835681    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 24 12:19:21.838065 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.835717    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 12:19:21.838065 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.835746    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 24 12:19:21.838065 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.836086    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 24 12:19:21.838639 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.838071    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nfjm7"]
Jul 24 12:19:21.838691 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.838652    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 12:19:21.838904 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.838860    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 24 12:19:21.839272 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.839003    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 24 12:19:21.839381 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.839361    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6vv5x\""
Jul 24 12:19:21.844686 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.844661    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl"]
Jul 24 12:19:21.844781 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.844703    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"]
Jul 24 12:19:21.844781 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.844723    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-847cc74b65-87hbw"]
Jul 24 12:19:21.844781 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.844741    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"]
Jul 24 12:19:21.844781 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.844764    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rxswx"]
Jul 24 12:19:21.844981 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.844796    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nfjm7"
Jul 24 12:19:21.847708 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.847688    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:21.848414 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.848380    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 12:19:21.848514 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.848424    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kcvkr\""
Jul 24 12:19:21.848584 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.848528    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 12:19:21.848584 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.848545    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 12:19:21.850279 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.850255    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 12:19:21.850783 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.850575    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 12:19:21.850876 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.850792    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-68vm8\""
Jul 24 12:19:21.855370 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.855350    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nfjm7"]
Jul 24 12:19:21.868512 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.868490    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rxswx"]
Jul 24 12:19:21.955475 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4f26b14d-d905-4635-b08b-85a486af1861-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dd55786f4-8c7xx\" (UID: \"4f26b14d-d905-4635-b08b-85a486af1861\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:21.955640 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72tpz\" (UniqueName: \"kubernetes.io/projected/0721c9c5-79f5-4e8f-94cb-1b095a68ebcf-kube-api-access-72tpz\") pod \"managed-serviceaccount-addon-agent-749f648c9d-vc8zl\" (UID: \"0721c9c5-79f5-4e8f-94cb-1b095a68ebcf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl"
Jul 24 12:19:21.955640 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955504    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjmx9\" (UniqueName: \"kubernetes.io/projected/d6224e74-72f6-4d5c-bbbd-d87bb40d9507-kube-api-access-gjmx9\") pod \"ingress-canary-nfjm7\" (UID: \"d6224e74-72f6-4d5c-bbbd-d87bb40d9507\") " pod="openshift-ingress-canary/ingress-canary-nfjm7"
Jul 24 12:19:21.955640 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94cc8f44-c20d-472c-a11d-986174072f55-config-volume\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:21.955640 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955618    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-installation-pull-secrets\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:21.955822 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-ca\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:21.955822 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsl7m\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-kube-api-access-wsl7m\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:21.955822 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-ca-trust-extracted\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:21.955822 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f26b14d-d905-4635-b08b-85a486af1861-tmp\") pod \"klusterlet-addon-workmgr-7dd55786f4-8c7xx\" (UID: \"4f26b14d-d905-4635-b08b-85a486af1861\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:21.955967 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:21.955967 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:21.955967 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955911    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-certificates\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:21.955967 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-bound-sa-token\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:21.955967 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.955962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/94cc8f44-c20d-472c-a11d-986174072f55-kube-api-access-9pdpl\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:21.956222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.956051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-image-registry-private-configuration\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:21.956222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.956087    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-trusted-ca\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:21.956222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.956115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94cc8f44-c20d-472c-a11d-986174072f55-tmp-dir\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:21.956222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.956141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-hub\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:21.956222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.956173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:21.956222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.956207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:21.956448 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.956252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmspv\" (UniqueName: \"kubernetes.io/projected/4f26b14d-d905-4635-b08b-85a486af1861-kube-api-access-dmspv\") pod \"klusterlet-addon-workmgr-7dd55786f4-8c7xx\" (UID: \"4f26b14d-d905-4635-b08b-85a486af1861\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:21.956448 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.956272    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0721c9c5-79f5-4e8f-94cb-1b095a68ebcf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-749f648c9d-vc8zl\" (UID: \"0721c9c5-79f5-4e8f-94cb-1b095a68ebcf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl"
Jul 24 12:19:21.956448 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.956291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6224e74-72f6-4d5c-bbbd-d87bb40d9507-cert\") pod \"ingress-canary-nfjm7\" (UID: \"d6224e74-72f6-4d5c-bbbd-d87bb40d9507\") " pod="openshift-ingress-canary/ingress-canary-nfjm7"
Jul 24 12:19:21.956448 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.956308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:21.956448 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:21.956326    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr6pm\" (UniqueName: \"kubernetes.io/projected/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-kube-api-access-nr6pm\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.057698 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.057611    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-installation-pull-secrets\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.057698 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.057653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-ca\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.057698 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.057693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsl7m\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-kube-api-access-wsl7m\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.057967 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.057722    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-ca-trust-extracted\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.057967 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.057754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f26b14d-d905-4635-b08b-85a486af1861-tmp\") pod \"klusterlet-addon-workmgr-7dd55786f4-8c7xx\" (UID: \"4f26b14d-d905-4635-b08b-85a486af1861\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:22.058116 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:22.058173 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.058173 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-certificates\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.058258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-bound-sa-token\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.058258 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:22.058197    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 12:19:22.058258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/94cc8f44-c20d-472c-a11d-986174072f55-kube-api-access-9pdpl\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:22.058258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-image-registry-private-configuration\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.058422 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:22.058262    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls podName:94cc8f44-c20d-472c-a11d-986174072f55 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:22.558241371 +0000 UTC m=+36.085988404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls") pod "dns-default-rxswx" (UID: "94cc8f44-c20d-472c-a11d-986174072f55") : secret "dns-default-metrics-tls" not found
Jul 24 12:19:22.058422 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-trusted-ca\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.058422 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94cc8f44-c20d-472c-a11d-986174072f55-tmp-dir\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:22.058422 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-hub\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.058422 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.058422 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.058742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmspv\" (UniqueName: \"kubernetes.io/projected/4f26b14d-d905-4635-b08b-85a486af1861-kube-api-access-dmspv\") pod \"klusterlet-addon-workmgr-7dd55786f4-8c7xx\" (UID: \"4f26b14d-d905-4635-b08b-85a486af1861\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:22.058742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0721c9c5-79f5-4e8f-94cb-1b095a68ebcf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-749f648c9d-vc8zl\" (UID: \"0721c9c5-79f5-4e8f-94cb-1b095a68ebcf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl"
Jul 24 12:19:22.058742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6224e74-72f6-4d5c-bbbd-d87bb40d9507-cert\") pod \"ingress-canary-nfjm7\" (UID: \"d6224e74-72f6-4d5c-bbbd-d87bb40d9507\") " pod="openshift-ingress-canary/ingress-canary-nfjm7"
Jul 24 12:19:22.058742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.058742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr6pm\" (UniqueName: \"kubernetes.io/projected/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-kube-api-access-nr6pm\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.058742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058572    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f26b14d-d905-4635-b08b-85a486af1861-tmp\") pod \"klusterlet-addon-workmgr-7dd55786f4-8c7xx\" (UID: \"4f26b14d-d905-4635-b08b-85a486af1861\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:22.058742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4f26b14d-d905-4635-b08b-85a486af1861-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dd55786f4-8c7xx\" (UID: \"4f26b14d-d905-4635-b08b-85a486af1861\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:22.058742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72tpz\" (UniqueName: \"kubernetes.io/projected/0721c9c5-79f5-4e8f-94cb-1b095a68ebcf-kube-api-access-72tpz\") pod \"managed-serviceaccount-addon-agent-749f648c9d-vc8zl\" (UID: \"0721c9c5-79f5-4e8f-94cb-1b095a68ebcf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl"
Jul 24 12:19:22.058742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjmx9\" (UniqueName: \"kubernetes.io/projected/d6224e74-72f6-4d5c-bbbd-d87bb40d9507-kube-api-access-gjmx9\") pod \"ingress-canary-nfjm7\" (UID: \"d6224e74-72f6-4d5c-bbbd-d87bb40d9507\") " pod="openshift-ingress-canary/ingress-canary-nfjm7"
Jul 24 12:19:22.058742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94cc8f44-c20d-472c-a11d-986174072f55-config-volume\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:22.058742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.058705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-ca-trust-extracted\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.059270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.059088    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-certificates\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.060730 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.060182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-trusted-ca\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.060730 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.060584    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94cc8f44-c20d-472c-a11d-986174072f55-config-volume\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:22.060929 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.060835    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94cc8f44-c20d-472c-a11d-986174072f55-tmp-dir\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:22.064747 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.064718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-image-registry-private-configuration\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.064747 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.064734    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4f26b14d-d905-4635-b08b-85a486af1861-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dd55786f4-8c7xx\" (UID: \"4f26b14d-d905-4635-b08b-85a486af1861\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:22.064991 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.064969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.065296 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:22.065275    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 12:19:22.065615 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:22.065298    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847cc74b65-87hbw: secret "image-registry-tls" not found
Jul 24 12:19:22.065615 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:22.065390    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls podName:a1b2b5c6-4978-40e1-ad9a-a39876b680a7 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:22.565371234 +0000 UTC m=+36.093118283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls") pod "image-registry-847cc74b65-87hbw" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7") : secret "image-registry-tls" not found
Jul 24 12:19:22.065615 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.065429    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.065789 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.065772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-ca\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.067501 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.067470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6224e74-72f6-4d5c-bbbd-d87bb40d9507-cert\") pod \"ingress-canary-nfjm7\" (UID: \"d6224e74-72f6-4d5c-bbbd-d87bb40d9507\") " pod="openshift-ingress-canary/ingress-canary-nfjm7"
Jul 24 12:19:22.067595 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.067580    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-installation-pull-secrets\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.068051 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.067999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.068130 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.068077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-hub\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.069899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.069877    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0721c9c5-79f5-4e8f-94cb-1b095a68ebcf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-749f648c9d-vc8zl\" (UID: \"0721c9c5-79f5-4e8f-94cb-1b095a68ebcf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl"
Jul 24 12:19:22.072437 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.072410    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsl7m\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-kube-api-access-wsl7m\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.074632 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.074557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmspv\" (UniqueName: \"kubernetes.io/projected/4f26b14d-d905-4635-b08b-85a486af1861-kube-api-access-dmspv\") pod \"klusterlet-addon-workmgr-7dd55786f4-8c7xx\" (UID: \"4f26b14d-d905-4635-b08b-85a486af1861\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:22.074632 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.074590    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72tpz\" (UniqueName: \"kubernetes.io/projected/0721c9c5-79f5-4e8f-94cb-1b095a68ebcf-kube-api-access-72tpz\") pod \"managed-serviceaccount-addon-agent-749f648c9d-vc8zl\" (UID: \"0721c9c5-79f5-4e8f-94cb-1b095a68ebcf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl"
Jul 24 12:19:22.075702 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.075683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/94cc8f44-c20d-472c-a11d-986174072f55-kube-api-access-9pdpl\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:22.075992 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.075968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-bound-sa-token\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.076197 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.076175    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr6pm\" (UniqueName: \"kubernetes.io/projected/362c3d1a-f1d4-45f8-9fbc-b91de7fc284d-kube-api-access-nr6pm\") pod \"cluster-proxy-proxy-agent-56bbb7fb48-x2kvx\" (UID: \"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.076281 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.076259    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjmx9\" (UniqueName: \"kubernetes.io/projected/d6224e74-72f6-4d5c-bbbd-d87bb40d9507-kube-api-access-gjmx9\") pod \"ingress-canary-nfjm7\" (UID: \"d6224e74-72f6-4d5c-bbbd-d87bb40d9507\") " pod="openshift-ingress-canary/ingress-canary-nfjm7"
Jul 24 12:19:22.156574 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.156543    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:22.168047 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.168008    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:19:22.168176 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.168040    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:22.176403 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.176235    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl"
Jul 24 12:19:22.206046 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.205847    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nfjm7"
Jul 24 12:19:22.265039 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.263880    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:22.365351 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.365283    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl"]
Jul 24 12:19:22.368862 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:19:22.368834    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0721c9c5_79f5_4e8f_94cb_1b095a68ebcf.slice/crio-2bf5b51a5717f105906752bf85cccf6d52d1244ebd199888f5dc034934568a6e WatchSource:0}: Error finding container 2bf5b51a5717f105906752bf85cccf6d52d1244ebd199888f5dc034934568a6e: Status 404 returned error can't find the container with id 2bf5b51a5717f105906752bf85cccf6d52d1244ebd199888f5dc034934568a6e
Jul 24 12:19:22.385166 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.385139    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"]
Jul 24 12:19:22.388592 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:19:22.388561    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod362c3d1a_f1d4_45f8_9fbc_b91de7fc284d.slice/crio-1b1d09b5ce4131c4a23fb93e3c136df824142b285d66fe10d327a8c6c2eb9f80 WatchSource:0}: Error finding container 1b1d09b5ce4131c4a23fb93e3c136df824142b285d66fe10d327a8c6c2eb9f80: Status 404 returned error can't find the container with id 1b1d09b5ce4131c4a23fb93e3c136df824142b285d66fe10d327a8c6c2eb9f80
Jul 24 12:19:22.390427 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.390361    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"]
Jul 24 12:19:22.392499 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:19:22.392478    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f26b14d_d905_4635_b08b_85a486af1861.slice/crio-49e5a84c9c033c6a2a25e7965b825397e8ee77b96681f4fd5dec376f956d656a WatchSource:0}: Error finding container 49e5a84c9c033c6a2a25e7965b825397e8ee77b96681f4fd5dec376f956d656a: Status 404 returned error can't find the container with id 49e5a84c9c033c6a2a25e7965b825397e8ee77b96681f4fd5dec376f956d656a
Jul 24 12:19:22.397997 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.397974    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nfjm7"]
Jul 24 12:19:22.400685 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:19:22.400656    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6224e74_72f6_4d5c_bbbd_d87bb40d9507.slice/crio-85ea995f1f64a947d9414429a854679b3ac901abe0b66b0a8d5772de852b9999 WatchSource:0}: Error finding container 85ea995f1f64a947d9414429a854679b3ac901abe0b66b0a8d5772de852b9999: Status 404 returned error can't find the container with id 85ea995f1f64a947d9414429a854679b3ac901abe0b66b0a8d5772de852b9999
Jul 24 12:19:22.564847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.564814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:22.565030 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:22.564967    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 12:19:22.565115 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:22.565046    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls podName:94cc8f44-c20d-472c-a11d-986174072f55 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:23.565004867 +0000 UTC m=+37.092751905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls") pod "dns-default-rxswx" (UID: "94cc8f44-c20d-472c-a11d-986174072f55") : secret "dns-default-metrics-tls" not found
Jul 24 12:19:22.665662 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.665573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:22.665811 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:22.665699    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 12:19:22.665811 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:22.665714    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847cc74b65-87hbw: secret "image-registry-tls" not found
Jul 24 12:19:22.665811 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:22.665779    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls podName:a1b2b5c6-4978-40e1-ad9a-a39876b680a7 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:23.665760095 +0000 UTC m=+37.193507130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls") pod "image-registry-847cc74b65-87hbw" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7") : secret "image-registry-tls" not found
Jul 24 12:19:22.971875 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.971145    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:22.971875 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.971147    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:22.971875 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.971286    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:22.974451 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.974426    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 12:19:22.974860 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.974841    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k4xp5\""
Jul 24 12:19:22.975088 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.975072    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 12:19:22.975364 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.975328    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 12:19:22.975553 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.975530    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zhxrp\""
Jul 24 12:19:22.975639 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:22.975555    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 12:19:23.130943 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:23.130872    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl" event={"ID":"0721c9c5-79f5-4e8f-94cb-1b095a68ebcf","Type":"ContainerStarted","Data":"2bf5b51a5717f105906752bf85cccf6d52d1244ebd199888f5dc034934568a6e"}
Jul 24 12:19:23.135693 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:23.135605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx" event={"ID":"4f26b14d-d905-4635-b08b-85a486af1861","Type":"ContainerStarted","Data":"49e5a84c9c033c6a2a25e7965b825397e8ee77b96681f4fd5dec376f956d656a"}
Jul 24 12:19:23.138252 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:23.138175    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nfjm7" event={"ID":"d6224e74-72f6-4d5c-bbbd-d87bb40d9507","Type":"ContainerStarted","Data":"85ea995f1f64a947d9414429a854679b3ac901abe0b66b0a8d5772de852b9999"}
Jul 24 12:19:23.141702 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:23.141643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx" event={"ID":"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d","Type":"ContainerStarted","Data":"1b1d09b5ce4131c4a23fb93e3c136df824142b285d66fe10d327a8c6c2eb9f80"}
Jul 24 12:19:23.573739 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:23.573667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:23.573907 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:23.573831    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 12:19:23.573907 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:23.573905    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls podName:94cc8f44-c20d-472c-a11d-986174072f55 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:25.573883044 +0000 UTC m=+39.101630094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls") pod "dns-default-rxswx" (UID: "94cc8f44-c20d-472c-a11d-986174072f55") : secret "dns-default-metrics-tls" not found
Jul 24 12:19:23.674435 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:23.674391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:23.674690 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:23.674671    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 12:19:23.674766 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:23.674695    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847cc74b65-87hbw: secret "image-registry-tls" not found
Jul 24 12:19:23.674766 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:23.674754    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls podName:a1b2b5c6-4978-40e1-ad9a-a39876b680a7 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:25.674734696 +0000 UTC m=+39.202481746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls") pod "image-registry-847cc74b65-87hbw" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7") : secret "image-registry-tls" not found
Jul 24 12:19:25.591882 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:25.591849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:25.592332 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:25.591994    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 12:19:25.592332 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:25.592095    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls podName:94cc8f44-c20d-472c-a11d-986174072f55 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:29.592073794 +0000 UTC m=+43.119820841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls") pod "dns-default-rxswx" (UID: "94cc8f44-c20d-472c-a11d-986174072f55") : secret "dns-default-metrics-tls" not found
Jul 24 12:19:25.692520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:25.692482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:25.692706 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:25.692659    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 12:19:25.692706 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:25.692682    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847cc74b65-87hbw: secret "image-registry-tls" not found
Jul 24 12:19:25.692818 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:25.692759    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls podName:a1b2b5c6-4978-40e1-ad9a-a39876b680a7 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:29.692737868 +0000 UTC m=+43.220484901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls") pod "image-registry-847cc74b65-87hbw" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7") : secret "image-registry-tls" not found
Jul 24 12:19:26.470493 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:26.470456    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:26.557933 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:26.557733    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:26.677044 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:26.676414    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:26.745084 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:26.744917    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:27.444030 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:27.443986    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:27.544369 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:27.544335    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:28.485979 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:28.485939    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:28.587929 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:28.587893    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:28.688860 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:28.688820    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:28.854069 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:28.854035    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:28.997329 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:28.997301    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:29.097034 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:29.096995    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:29.197713 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:29.197630    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:29.420661 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:29.420624    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:29.517296 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:29.517258    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:29.623906 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:29.623869    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:29.624081 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:29.624062    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 12:19:29.624151 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:29.624137    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls podName:94cc8f44-c20d-472c-a11d-986174072f55 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:37.624116343 +0000 UTC m=+51.151863390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls") pod "dns-default-rxswx" (UID: "94cc8f44-c20d-472c-a11d-986174072f55") : secret "dns-default-metrics-tls" not found
Jul 24 12:19:29.725096 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:29.725062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:29.725271 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:29.725215    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 12:19:29.725271 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:29.725236    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847cc74b65-87hbw: secret "image-registry-tls" not found
Jul 24 12:19:29.725369 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:29.725309    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls podName:a1b2b5c6-4978-40e1-ad9a-a39876b680a7 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:37.725290826 +0000 UTC m=+51.253037861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls") pod "image-registry-847cc74b65-87hbw" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7") : secret "image-registry-tls" not found
Jul 24 12:19:29.814848 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:29.814771    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:29.870982 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:29.870946    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:30.540134 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.540093    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"]
Jul 24 12:19:30.559932 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.559905    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"]
Jul 24 12:19:30.560322 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.560300    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"
Jul 24 12:19:30.565738 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.565425    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.565738 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.565599    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 24 12:19:30.565911 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.565814    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.565992 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.565974    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-svmtt\""
Jul 24 12:19:30.566110 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.566096    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 24 12:19:30.574062 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.574041    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-zlm49"]
Jul 24 12:19:30.574183 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.574115    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:19:30.577246 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.577218    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 24 12:19:30.577389 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.577257    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.577389 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.577263    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.577468 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.577424    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xvs99\""
Jul 24 12:19:30.594255 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.594230    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"]
Jul 24 12:19:30.595411 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.595112    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.598380 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.598359    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 24 12:19:30.598681 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.598631    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.598681 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.598644    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 24 12:19:30.599105 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.599086    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.599186 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.599118    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vw82t\""
Jul 24 12:19:30.603855 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.603838    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 24 12:19:30.616296 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.616275    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"]
Jul 24 12:19:30.616397 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.616315    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"]
Jul 24 12:19:30.616397 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.616345    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"]
Jul 24 12:19:30.616397 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.616364    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-zlm49"]
Jul 24 12:19:30.616397 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.616387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:30.619420 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.619403    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.619516 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.619417    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 24 12:19:30.619516 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.619483    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-c287l\""
Jul 24 12:19:30.619626 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.619423    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.621365 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.621345    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 24 12:19:30.640804 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.640781    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-4vl9k"]
Jul 24 12:19:30.658492 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.658470    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"]
Jul 24 12:19:30.658926 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.658907    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4vl9k"
Jul 24 12:19:30.661900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.661881    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-l9ndj\""
Jul 24 12:19:30.676471 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.676453    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-rfts5"]
Jul 24 12:19:30.676608 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.676590    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"
Jul 24 12:19:30.681305 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.681286    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.681409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.681339    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 24 12:19:30.685867 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.684754    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.685867 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.684843    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2q5n5\""
Jul 24 12:19:30.685867 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.685196    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 24 12:19:30.696651 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.696631    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-4vl9k"]
Jul 24 12:19:30.696767 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.696665    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-86b5bcffdf-j8mkb"]
Jul 24 12:19:30.696827 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.696763    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.700386 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.700366    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wx456\""
Jul 24 12:19:30.701062 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.701042    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 24 12:19:30.701270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.701247    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.701820 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.701803    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.701939 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.701918    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 24 12:19:30.705721 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.705703    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 24 12:19:30.716597 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.716578    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-rfts5"]
Jul 24 12:19:30.716702 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.716604    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"]
Jul 24 12:19:30.716702 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.716624    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86b5bcffdf-j8mkb"]
Jul 24 12:19:30.716793 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.716714    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.719512 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.719479    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 24 12:19:30.719714 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.719638    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.719714 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.719641    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 24 12:19:30.719714 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.719701    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 24 12:19:30.719878 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.719721    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.719878 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.719848    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4lmzg\""
Jul 24 12:19:30.720109 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.720092    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 24 12:19:30.732429 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:30.732545 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vns8t\" (UniqueName: \"kubernetes.io/projected/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-kube-api-access-vns8t\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:30.732545 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c42a44d-e70d-4baa-8fd9-5489b77c1e18-config\") pod \"service-ca-operator-5d6c75f84f-sdlcq\" (UID: \"0c42a44d-e70d-4baa-8fd9-5489b77c1e18\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"
Jul 24 12:19:30.732545 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732492    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:30.732545 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zz7\" (UniqueName: \"kubernetes.io/projected/0d55faa5-7be5-4ff1-b37e-4c556baaa14e-kube-api-access-46zz7\") pod \"network-check-source-7d88cb55d8-4vl9k\" (UID: \"0d55faa5-7be5-4ff1-b37e-4c556baaa14e\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4vl9k"
Jul 24 12:19:30.732738 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-tmp\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.732738 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-serving-cert\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.732738 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732700    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkw4x\" (UniqueName: \"kubernetes.io/projected/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-kube-api-access-xkw4x\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.732738 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f9bc\" (UniqueName: \"kubernetes.io/projected/0c42a44d-e70d-4baa-8fd9-5489b77c1e18-kube-api-access-6f9bc\") pod \"service-ca-operator-5d6c75f84f-sdlcq\" (UID: \"0c42a44d-e70d-4baa-8fd9-5489b77c1e18\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"
Jul 24 12:19:30.732913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732766    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-snapshots\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.732913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.732913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c42a44d-e70d-4baa-8fd9-5489b77c1e18-serving-cert\") pod \"service-ca-operator-5d6c75f84f-sdlcq\" (UID: \"0c42a44d-e70d-4baa-8fd9-5489b77c1e18\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"
Jul 24 12:19:30.732913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-d4jq6\" (UID: \"0d88bedb-dc43-4a7d-8a6c-375525c08823\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:19:30.733114 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-service-ca-bundle\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.733114 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.732967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6p9b\" (UniqueName: \"kubernetes.io/projected/0d88bedb-dc43-4a7d-8a6c-375525c08823-kube-api-access-l6p9b\") pod \"cluster-samples-operator-6748489656-d4jq6\" (UID: \"0d88bedb-dc43-4a7d-8a6c-375525c08823\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:19:30.741278 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.741256    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-k6xvq"]
Jul 24 12:19:30.759111 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.759087    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-k6xvq"]
Jul 24 12:19:30.759236 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.759197    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-k6xvq"
Jul 24 12:19:30.762344 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.762324    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.762902 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.762883    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.762995 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.762884    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9hcb2\""
Jul 24 12:19:30.833640 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.833566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkw4x\" (UniqueName: \"kubernetes.io/projected/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-kube-api-access-xkw4x\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.833640 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.833605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e03b0875-5355-4282-96e5-efe11ccd7618-serving-cert\") pod \"console-operator-584dcb6b54-rfts5\" (UID: \"e03b0875-5355-4282-96e5-efe11ccd7618\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.833640 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.833635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f9bc\" (UniqueName: \"kubernetes.io/projected/0c42a44d-e70d-4baa-8fd9-5489b77c1e18-kube-api-access-6f9bc\") pod \"service-ca-operator-5d6c75f84f-sdlcq\" (UID: \"0c42a44d-e70d-4baa-8fd9-5489b77c1e18\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"
Jul 24 12:19:30.833870 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.833676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-snapshots\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.833870 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.833698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.833870 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.833725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e03b0875-5355-4282-96e5-efe11ccd7618-config\") pod \"console-operator-584dcb6b54-rfts5\" (UID: \"e03b0875-5355-4282-96e5-efe11ccd7618\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.833870 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.833793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-default-certificate\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.833870 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.833827    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.834133 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.833870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c42a44d-e70d-4baa-8fd9-5489b77c1e18-serving-cert\") pod \"service-ca-operator-5d6c75f84f-sdlcq\" (UID: \"0c42a44d-e70d-4baa-8fd9-5489b77c1e18\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"
Jul 24 12:19:30.834133 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.833896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f810f78d-7227-4bd0-aab4-194ced389280-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n\" (UID: \"f810f78d-7227-4bd0-aab4-194ced389280\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"
Jul 24 12:19:30.834133 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.833943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-d4jq6\" (UID: \"0d88bedb-dc43-4a7d-8a6c-375525c08823\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:19:30.834133 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.833969    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f810f78d-7227-4bd0-aab4-194ced389280-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n\" (UID: \"f810f78d-7227-4bd0-aab4-194ced389280\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"
Jul 24 12:19:30.834133 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834001    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.834133 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:30.834046    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 12:19:30.834133 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e03b0875-5355-4282-96e5-efe11ccd7618-trusted-ca\") pod \"console-operator-584dcb6b54-rfts5\" (UID: \"e03b0875-5355-4282-96e5-efe11ccd7618\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.834133 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-service-ca-bundle\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.834133 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:30.834107    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls podName:0d88bedb-dc43-4a7d-8a6c-375525c08823 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:31.334088546 +0000 UTC m=+44.861835578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls") pod "cluster-samples-operator-6748489656-d4jq6" (UID: "0d88bedb-dc43-4a7d-8a6c-375525c08823") : secret "samples-operator-tls" not found
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6p9b\" (UniqueName: \"kubernetes.io/projected/0d88bedb-dc43-4a7d-8a6c-375525c08823-kube-api-access-l6p9b\") pod \"cluster-samples-operator-6748489656-d4jq6\" (UID: \"0d88bedb-dc43-4a7d-8a6c-375525c08823\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4gtk\" (UniqueName: \"kubernetes.io/projected/7dcef893-f48b-41d9-acd1-50636e761528-kube-api-access-w4gtk\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vns8t\" (UniqueName: \"kubernetes.io/projected/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-kube-api-access-vns8t\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834261    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-stats-auth\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c42a44d-e70d-4baa-8fd9-5489b77c1e18-config\") pod \"service-ca-operator-5d6c75f84f-sdlcq\" (UID: \"0c42a44d-e70d-4baa-8fd9-5489b77c1e18\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xxs\" (UniqueName: \"kubernetes.io/projected/e03b0875-5355-4282-96e5-efe11ccd7618-kube-api-access-v7xxs\") pod \"console-operator-584dcb6b54-rfts5\" (UID: \"e03b0875-5355-4282-96e5-efe11ccd7618\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zz7\" (UniqueName: \"kubernetes.io/projected/0d55faa5-7be5-4ff1-b37e-4c556baaa14e-kube-api-access-46zz7\") pod \"network-check-source-7d88cb55d8-4vl9k\" (UID: \"0d55faa5-7be5-4ff1-b37e-4c556baaa14e\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4vl9k"
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-tmp\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-serving-cert\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834504    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmxjj\" (UniqueName: \"kubernetes.io/projected/0c0e4621-d282-4a07-b820-9df31d6ce258-kube-api-access-wmxjj\") pod \"volume-data-source-validator-549c6d54cf-k6xvq\" (UID: \"0c0e4621-d282-4a07-b820-9df31d6ce258\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-k6xvq"
Jul 24 12:19:30.834533 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:30.834532    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 12:19:30.835430 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:30.834575    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls podName:4dcb8aa8-6eb1-4922-be33-3969dbe5ea51 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:31.334560155 +0000 UTC m=+44.862307187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jlrvs" (UID: "4dcb8aa8-6eb1-4922-be33-3969dbe5ea51") : secret "cluster-monitoring-operator-tls" not found
Jul 24 12:19:30.835430 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vm2q\" (UniqueName: \"kubernetes.io/projected/f810f78d-7227-4bd0-aab4-194ced389280-kube-api-access-5vm2q\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n\" (UID: \"f810f78d-7227-4bd0-aab4-194ced389280\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"
Jul 24 12:19:30.835430 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834597    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-snapshots\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.835430 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834692    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.835430 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c42a44d-e70d-4baa-8fd9-5489b77c1e18-config\") pod \"service-ca-operator-5d6c75f84f-sdlcq\" (UID: \"0c42a44d-e70d-4baa-8fd9-5489b77c1e18\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"
Jul 24 12:19:30.835430 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:30.835430 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834971    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-tmp\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.835430 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.834983    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-service-ca-bundle\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.837205 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.837179    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c42a44d-e70d-4baa-8fd9-5489b77c1e18-serving-cert\") pod \"service-ca-operator-5d6c75f84f-sdlcq\" (UID: \"0c42a44d-e70d-4baa-8fd9-5489b77c1e18\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"
Jul 24 12:19:30.837306 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.837247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-serving-cert\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.844472 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.844449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vns8t\" (UniqueName: \"kubernetes.io/projected/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-kube-api-access-vns8t\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:30.844892 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.844865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkw4x\" (UniqueName: \"kubernetes.io/projected/45ae8a84-d47e-461b-bd9a-6d076a5b94f0-kube-api-access-xkw4x\") pod \"insights-operator-5885fdb759-zlm49\" (UID: \"45ae8a84-d47e-461b-bd9a-6d076a5b94f0\") " pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.844997 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.844976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f9bc\" (UniqueName: \"kubernetes.io/projected/0c42a44d-e70d-4baa-8fd9-5489b77c1e18-kube-api-access-6f9bc\") pod \"service-ca-operator-5d6c75f84f-sdlcq\" (UID: \"0c42a44d-e70d-4baa-8fd9-5489b77c1e18\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"
Jul 24 12:19:30.845091 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.845064    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6p9b\" (UniqueName: \"kubernetes.io/projected/0d88bedb-dc43-4a7d-8a6c-375525c08823-kube-api-access-l6p9b\") pod \"cluster-samples-operator-6748489656-d4jq6\" (UID: \"0d88bedb-dc43-4a7d-8a6c-375525c08823\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:19:30.846865 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.846840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zz7\" (UniqueName: \"kubernetes.io/projected/0d55faa5-7be5-4ff1-b37e-4c556baaa14e-kube-api-access-46zz7\") pod \"network-check-source-7d88cb55d8-4vl9k\" (UID: \"0d55faa5-7be5-4ff1-b37e-4c556baaa14e\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4vl9k"
Jul 24 12:19:30.847555 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.847528    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:30.872940 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.872916    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"
Jul 24 12:19:30.906425 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.906409    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-zlm49"
Jul 24 12:19:30.935459 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e03b0875-5355-4282-96e5-efe11ccd7618-serving-cert\") pod \"console-operator-584dcb6b54-rfts5\" (UID: \"e03b0875-5355-4282-96e5-efe11ccd7618\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.935538 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e03b0875-5355-4282-96e5-efe11ccd7618-config\") pod \"console-operator-584dcb6b54-rfts5\" (UID: \"e03b0875-5355-4282-96e5-efe11ccd7618\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.935538 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-default-certificate\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.935538 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935523    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.935659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f810f78d-7227-4bd0-aab4-194ced389280-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n\" (UID: \"f810f78d-7227-4bd0-aab4-194ced389280\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"
Jul 24 12:19:30.935659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f810f78d-7227-4bd0-aab4-194ced389280-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n\" (UID: \"f810f78d-7227-4bd0-aab4-194ced389280\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"
Jul 24 12:19:30.935659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935626    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.935805 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e03b0875-5355-4282-96e5-efe11ccd7618-trusted-ca\") pod \"console-operator-584dcb6b54-rfts5\" (UID: \"e03b0875-5355-4282-96e5-efe11ccd7618\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.935805 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:30.935674    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 12:19:30.935805 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gtk\" (UniqueName: \"kubernetes.io/projected/7dcef893-f48b-41d9-acd1-50636e761528-kube-api-access-w4gtk\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.935805 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:30.935753    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs podName:7dcef893-f48b-41d9-acd1-50636e761528 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:31.435731523 +0000 UTC m=+44.963478559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs") pod "router-default-86b5bcffdf-j8mkb" (UID: "7dcef893-f48b-41d9-acd1-50636e761528") : secret "router-metrics-certs-default" not found
Jul 24 12:19:30.935805 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-stats-auth\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.936163 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:30.935863    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle podName:7dcef893-f48b-41d9-acd1-50636e761528 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:31.435843581 +0000 UTC m=+44.963590637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle") pod "router-default-86b5bcffdf-j8mkb" (UID: "7dcef893-f48b-41d9-acd1-50636e761528") : configmap references non-existent config key: service-ca.crt
Jul 24 12:19:30.936163 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xxs\" (UniqueName: \"kubernetes.io/projected/e03b0875-5355-4282-96e5-efe11ccd7618-kube-api-access-v7xxs\") pod \"console-operator-584dcb6b54-rfts5\" (UID: \"e03b0875-5355-4282-96e5-efe11ccd7618\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.936163 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmxjj\" (UniqueName: \"kubernetes.io/projected/0c0e4621-d282-4a07-b820-9df31d6ce258-kube-api-access-wmxjj\") pod \"volume-data-source-validator-549c6d54cf-k6xvq\" (UID: \"0c0e4621-d282-4a07-b820-9df31d6ce258\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-k6xvq"
Jul 24 12:19:30.936163 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.935979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vm2q\" (UniqueName: \"kubernetes.io/projected/f810f78d-7227-4bd0-aab4-194ced389280-kube-api-access-5vm2q\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n\" (UID: \"f810f78d-7227-4bd0-aab4-194ced389280\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"
Jul 24 12:19:30.936401 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.936286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e03b0875-5355-4282-96e5-efe11ccd7618-config\") pod \"console-operator-584dcb6b54-rfts5\" (UID: \"e03b0875-5355-4282-96e5-efe11ccd7618\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.936609 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.936584    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f810f78d-7227-4bd0-aab4-194ced389280-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n\" (UID: \"f810f78d-7227-4bd0-aab4-194ced389280\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"
Jul 24 12:19:30.936925 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.936907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e03b0875-5355-4282-96e5-efe11ccd7618-trusted-ca\") pod \"console-operator-584dcb6b54-rfts5\" (UID: \"e03b0875-5355-4282-96e5-efe11ccd7618\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.938250 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.938232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-default-certificate\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.938250 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.938252    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-stats-auth\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.938585 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.938568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e03b0875-5355-4282-96e5-efe11ccd7618-serving-cert\") pod \"console-operator-584dcb6b54-rfts5\" (UID: \"e03b0875-5355-4282-96e5-efe11ccd7618\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.938700 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.938685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f810f78d-7227-4bd0-aab4-194ced389280-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n\" (UID: \"f810f78d-7227-4bd0-aab4-194ced389280\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"
Jul 24 12:19:30.948532 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.948506    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vm2q\" (UniqueName: \"kubernetes.io/projected/f810f78d-7227-4bd0-aab4-194ced389280-kube-api-access-5vm2q\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n\" (UID: \"f810f78d-7227-4bd0-aab4-194ced389280\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"
Jul 24 12:19:30.949486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.949419    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:30.949627 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.949600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xxs\" (UniqueName: \"kubernetes.io/projected/e03b0875-5355-4282-96e5-efe11ccd7618-kube-api-access-v7xxs\") pod \"console-operator-584dcb6b54-rfts5\" (UID: \"e03b0875-5355-4282-96e5-efe11ccd7618\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:30.949888 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.949871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4gtk\" (UniqueName: \"kubernetes.io/projected/7dcef893-f48b-41d9-acd1-50636e761528-kube-api-access-w4gtk\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:30.950100 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.950079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmxjj\" (UniqueName: \"kubernetes.io/projected/0c0e4621-d282-4a07-b820-9df31d6ce258-kube-api-access-wmxjj\") pod \"volume-data-source-validator-549c6d54cf-k6xvq\" (UID: \"0c0e4621-d282-4a07-b820-9df31d6ce258\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-k6xvq"
Jul 24 12:19:30.971358 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.971339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4vl9k"
Jul 24 12:19:30.988397 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:30.988378    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"
Jul 24 12:19:31.007133 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.007111    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:31.070181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.070153    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-k6xvq"
Jul 24 12:19:31.339460 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.339425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-d4jq6\" (UID: \"0d88bedb-dc43-4a7d-8a6c-375525c08823\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:19:31.339600 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.339493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:31.339600 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:31.339577    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 12:19:31.339667 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:31.339641    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls podName:0d88bedb-dc43-4a7d-8a6c-375525c08823 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:32.339625482 +0000 UTC m=+45.867372533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls") pod "cluster-samples-operator-6748489656-d4jq6" (UID: "0d88bedb-dc43-4a7d-8a6c-375525c08823") : secret "samples-operator-tls" not found
Jul 24 12:19:31.339706 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:31.339580    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 12:19:31.339742 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:31.339718    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls podName:4dcb8aa8-6eb1-4922-be33-3969dbe5ea51 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:32.339706699 +0000 UTC m=+45.867453731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jlrvs" (UID: "4dcb8aa8-6eb1-4922-be33-3969dbe5ea51") : secret "cluster-monitoring-operator-tls" not found
Jul 24 12:19:31.400773 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.400687    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:31.440760 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.440732    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:31.440889 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.440795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:31.440889 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:31.440864    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 12:19:31.441006 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:31.440922    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs podName:7dcef893-f48b-41d9-acd1-50636e761528 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:32.44090735 +0000 UTC m=+45.968654383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs") pod "router-default-86b5bcffdf-j8mkb" (UID: "7dcef893-f48b-41d9-acd1-50636e761528") : secret "router-metrics-certs-default" not found
Jul 24 12:19:31.441006 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:31.440936    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle podName:7dcef893-f48b-41d9-acd1-50636e761528 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:32.440930087 +0000 UTC m=+45.968677118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle") pod "router-default-86b5bcffdf-j8mkb" (UID: "7dcef893-f48b-41d9-acd1-50636e761528") : configmap references non-existent config key: service-ca.crt
Jul 24 12:19:31.732313 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.730588    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-4vl9k"]
Jul 24 12:19:31.742459 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:19:31.742145    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d55faa5_7be5_4ff1_b37e_4c556baaa14e.slice/crio-0465cc4201d83b9d8c46daf55a9528473c737d0dfed5dc1487e69a7d81d14963 WatchSource:0}: Error finding container 0465cc4201d83b9d8c46daf55a9528473c737d0dfed5dc1487e69a7d81d14963: Status 404 returned error can't find the container with id 0465cc4201d83b9d8c46daf55a9528473c737d0dfed5dc1487e69a7d81d14963
Jul 24 12:19:31.748839 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.748815    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-rfts5"]
Jul 24 12:19:31.765890 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.765801    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n"]
Jul 24 12:19:31.795904 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.795806    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-zlm49"]
Jul 24 12:19:31.798230 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:19:31.798181    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ae8a84_d47e_461b_bd9a_6d076a5b94f0.slice/crio-ba486553a80e8a855c303c13b0aecf304d3d7bd2d21dd3f4e1d29478e510b030 WatchSource:0}: Error finding container ba486553a80e8a855c303c13b0aecf304d3d7bd2d21dd3f4e1d29478e510b030: Status 404 returned error can't find the container with id ba486553a80e8a855c303c13b0aecf304d3d7bd2d21dd3f4e1d29478e510b030
Jul 24 12:19:31.803010 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.802732    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-k6xvq"]
Jul 24 12:19:31.811822 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:31.811304    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq"]
Jul 24 12:19:31.820636 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:19:31.820612    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c42a44d_e70d_4baa_8fd9_5489b77c1e18.slice/crio-dce846c5b89a7f78d218de6e5c2cd345369b4c517ce41c8eb0c87aaf43982f79 WatchSource:0}: Error finding container dce846c5b89a7f78d218de6e5c2cd345369b4c517ce41c8eb0c87aaf43982f79: Status 404 returned error can't find the container with id dce846c5b89a7f78d218de6e5c2cd345369b4c517ce41c8eb0c87aaf43982f79
Jul 24 12:19:32.167641 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.167555    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx" event={"ID":"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d","Type":"ContainerStarted","Data":"c49ee9c73fb4a572e1754c6810c398049144fdbd5be80dee4c4e1edcdb27893d"}
Jul 24 12:19:32.167641 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.167604    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx" event={"ID":"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d","Type":"ContainerStarted","Data":"c6c77491742c83b3d22902e722cab096008339f30f09fdf8c4441d9c8feb56f7"}
Jul 24 12:19:32.167641 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.167624    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx" event={"ID":"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d","Type":"ContainerStarted","Data":"59d5ba1027708731503cb0409f33f72c597a57819f652f6438cacb7afbea9e88"}
Jul 24 12:19:32.168997 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.168974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl" event={"ID":"0721c9c5-79f5-4e8f-94cb-1b095a68ebcf","Type":"ContainerStarted","Data":"b413720ca5f2088f4dd72154e19bb3668eed9a60f25dd874dcd29f8ab58411e5"}
Jul 24 12:19:32.171609 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.171585    2566 generic.go:356] "Generic (PLEG): container finished" podID="32e6a712-727d-4045-927a-dd28b210caba" containerID="d2397fc46316f980d62084a1a8152babd996464c031959fa95396915a166403b" exitCode=0
Jul 24 12:19:32.171712 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.171651    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm5x2" event={"ID":"32e6a712-727d-4045-927a-dd28b210caba","Type":"ContainerDied","Data":"d2397fc46316f980d62084a1a8152babd996464c031959fa95396915a166403b"}
Jul 24 12:19:32.173181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.173155    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx" event={"ID":"4f26b14d-d905-4635-b08b-85a486af1861","Type":"ContainerStarted","Data":"fc98ae2dffafa702bc67d4baf8a18f57a5bcb1ed94b55f111ed932a24521b7d5"}
Jul 24 12:19:32.173298 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.173204    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:32.174617 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.174590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-k6xvq" event={"ID":"0c0e4621-d282-4a07-b820-9df31d6ce258","Type":"ContainerStarted","Data":"a667ae7818bb9b4ea77de48377ffdb4cb976cd9911c0360603aed04fdd4cb7f0"}
Jul 24 12:19:32.174927 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.174909    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx"
Jul 24 12:19:32.175666 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.175649    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq" event={"ID":"0c42a44d-e70d-4baa-8fd9-5489b77c1e18","Type":"ContainerStarted","Data":"dce846c5b89a7f78d218de6e5c2cd345369b4c517ce41c8eb0c87aaf43982f79"}
Jul 24 12:19:32.176681 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.176661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n" event={"ID":"f810f78d-7227-4bd0-aab4-194ced389280","Type":"ContainerStarted","Data":"f4e87aea75aa1086eeb2acfb5e8d758fe29496d72cb00102ce14f06bbaa013e5"}
Jul 24 12:19:32.177616 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.177598    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zlm49" event={"ID":"45ae8a84-d47e-461b-bd9a-6d076a5b94f0","Type":"ContainerStarted","Data":"ba486553a80e8a855c303c13b0aecf304d3d7bd2d21dd3f4e1d29478e510b030"}
Jul 24 12:19:32.178935 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.178915    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nfjm7" event={"ID":"d6224e74-72f6-4d5c-bbbd-d87bb40d9507","Type":"ContainerStarted","Data":"98fefed096a54c90d4aaac1bc68ca05c8bab38abc95c10fa4821a5c059b6870e"}
Jul 24 12:19:32.179971 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.179953    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4vl9k" event={"ID":"0d55faa5-7be5-4ff1-b37e-4c556baaa14e","Type":"ContainerStarted","Data":"0465cc4201d83b9d8c46daf55a9528473c737d0dfed5dc1487e69a7d81d14963"}
Jul 24 12:19:32.180930 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.180911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5" event={"ID":"e03b0875-5355-4282-96e5-efe11ccd7618","Type":"ContainerStarted","Data":"30287c29e106f302afa4ba28005d853fd352d1a6cac1292de0910f7ac353022a"}
Jul 24 12:19:32.189527 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.189484    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx" podStartSLOduration=26.091466466 podStartE2EDuration="35.189470272s" totalImagesPullingTime="9.098003806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:32.189399516 +0000 UTC m=+45.717146570" watchObservedRunningTime="2026-07-24 12:19:32.189470272 +0000 UTC m=+45.717217325"
Jul 24 12:19:32.207886 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.207839    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nfjm7" podStartSLOduration=2.127140657 podStartE2EDuration="11.207821041s" totalImagesPullingTime="9.080680384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:32.206662599 +0000 UTC m=+45.734409660" watchObservedRunningTime="2026-07-24 12:19:32.207821041 +0000 UTC m=+45.735568096"
Jul 24 12:19:32.253961 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.253916    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dd55786f4-8c7xx" podStartSLOduration=26.146780732 podStartE2EDuration="35.25389925s" totalImagesPullingTime="9.107118518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:32.253328101 +0000 UTC m=+45.781075157" watchObservedRunningTime="2026-07-24 12:19:32.25389925 +0000 UTC m=+45.781646282"
Jul 24 12:19:32.278642 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.278603    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749f648c9d-vc8zl" podStartSLOduration=26.163143518 podStartE2EDuration="35.278585829s" totalImagesPullingTime="9.115442311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:32.278414104 +0000 UTC m=+45.806161159" watchObservedRunningTime="2026-07-24 12:19:32.278585829 +0000 UTC m=+45.806332883"
Jul 24 12:19:32.359686 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.359654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-d4jq6\" (UID: \"0d88bedb-dc43-4a7d-8a6c-375525c08823\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:19:32.359797 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.359756    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:32.359945 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:32.359889    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 12:19:32.360013 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:32.359949    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls podName:4dcb8aa8-6eb1-4922-be33-3969dbe5ea51 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:34.359932294 +0000 UTC m=+47.887679326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jlrvs" (UID: "4dcb8aa8-6eb1-4922-be33-3969dbe5ea51") : secret "cluster-monitoring-operator-tls" not found
Jul 24 12:19:32.360101 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:32.360062    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 12:19:32.360135 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:32.360118    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls podName:0d88bedb-dc43-4a7d-8a6c-375525c08823 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:34.360100946 +0000 UTC m=+47.887847978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls") pod "cluster-samples-operator-6748489656-d4jq6" (UID: "0d88bedb-dc43-4a7d-8a6c-375525c08823") : secret "samples-operator-tls" not found
Jul 24 12:19:32.460974 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.460888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:32.460974 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:32.460961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:32.461208 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:32.461194    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle podName:7dcef893-f48b-41d9-acd1-50636e761528 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:34.461175316 +0000 UTC m=+47.988922350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle") pod "router-default-86b5bcffdf-j8mkb" (UID: "7dcef893-f48b-41d9-acd1-50636e761528") : configmap references non-existent config key: service-ca.crt
Jul 24 12:19:32.461542 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:32.461521    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 12:19:32.461637 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:32.461574    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs podName:7dcef893-f48b-41d9-acd1-50636e761528 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:34.461559176 +0000 UTC m=+47.989306207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs") pod "router-default-86b5bcffdf-j8mkb" (UID: "7dcef893-f48b-41d9-acd1-50636e761528") : secret "router-metrics-certs-default" not found
Jul 24 12:19:33.191595 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:33.188989    2566 generic.go:356] "Generic (PLEG): container finished" podID="32e6a712-727d-4045-927a-dd28b210caba" containerID="ec3283467fcc1c943019bf40f9c303c7d22e57f15be3df8d9a755d0214e2dd16" exitCode=0
Jul 24 12:19:33.191595 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:33.190481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm5x2" event={"ID":"32e6a712-727d-4045-927a-dd28b210caba","Type":"ContainerDied","Data":"ec3283467fcc1c943019bf40f9c303c7d22e57f15be3df8d9a755d0214e2dd16"}
Jul 24 12:19:34.199773 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:34.199729    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm5x2" event={"ID":"32e6a712-727d-4045-927a-dd28b210caba","Type":"ContainerStarted","Data":"f4911641ed9df99294207748b89461a2e06aa83ed607c4c45e130e0cbb46ff22"}
Jul 24 12:19:34.202413 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:34.202378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-k6xvq" event={"ID":"0c0e4621-d282-4a07-b820-9df31d6ce258","Type":"ContainerStarted","Data":"f4a75300f9324b24e9ea24839b868a2c71a04f606b28adbfb291cdfacda43970"}
Jul 24 12:19:34.228613 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:34.228555    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mm5x2" podStartSLOduration=9.197413075 podStartE2EDuration="47.228533816s" totalImagesPullingTime="38.031120741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:34.225401647 +0000 UTC m=+47.753148736" watchObservedRunningTime="2026-07-24 12:19:34.228533816 +0000 UTC m=+47.756280870"
Jul 24 12:19:34.246365 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:34.246311    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-k6xvq" podStartSLOduration=2.450317702 podStartE2EDuration="4.24629387s" totalImagesPullingTime="1.795976168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:34.245820652 +0000 UTC m=+47.773567708" watchObservedRunningTime="2026-07-24 12:19:34.24629387 +0000 UTC m=+47.774040902"
Jul 24 12:19:34.383827 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:34.383791    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:34.386722 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:34.386693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-d4jq6\" (UID: \"0d88bedb-dc43-4a7d-8a6c-375525c08823\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:19:34.386858 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:34.386840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:34.386960 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:34.386943    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 12:19:34.387038 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:34.387011    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls podName:4dcb8aa8-6eb1-4922-be33-3969dbe5ea51 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:38.386992137 +0000 UTC m=+51.914739168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jlrvs" (UID: "4dcb8aa8-6eb1-4922-be33-3969dbe5ea51") : secret "cluster-monitoring-operator-tls" not found
Jul 24 12:19:34.387967 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:34.387945    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 12:19:34.388114 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:34.388005    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls podName:0d88bedb-dc43-4a7d-8a6c-375525c08823 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:38.387988138 +0000 UTC m=+51.915735172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls") pod "cluster-samples-operator-6748489656-d4jq6" (UID: "0d88bedb-dc43-4a7d-8a6c-375525c08823") : secret "samples-operator-tls" not found
Jul 24 12:19:34.481132 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:34.481051    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:34.487649 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:34.487617    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:34.487802 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:34.487703    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:34.487802 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:34.487779    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 12:19:34.487880 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:34.487857    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs podName:7dcef893-f48b-41d9-acd1-50636e761528 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:38.487827272 +0000 UTC m=+52.015574318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs") pod "router-default-86b5bcffdf-j8mkb" (UID: "7dcef893-f48b-41d9-acd1-50636e761528") : secret "router-metrics-certs-default" not found
Jul 24 12:19:34.487880 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:34.487876    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle podName:7dcef893-f48b-41d9-acd1-50636e761528 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:38.487868055 +0000 UTC m=+52.015615088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle") pod "router-default-86b5bcffdf-j8mkb" (UID: "7dcef893-f48b-41d9-acd1-50636e761528") : configmap references non-existent config key: service-ca.crt
Jul 24 12:19:34.586182 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:34.586144    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:35.098234 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:35.098185    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:35.198714 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:35.198678    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:37.376036 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.375995    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-z598v"]
Jul 24 12:19:37.395517 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.395485    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-z598v"]
Jul 24 12:19:37.395703 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.395605    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:19:37.398657 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.398632    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 24 12:19:37.398835 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.398703    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 24 12:19:37.399058 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.399042    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tz7fl\""
Jul 24 12:19:37.413932 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.413905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d0d7097-31bb-4535-96bc-c30b51af5567-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-z598v\" (UID: \"1d0d7097-31bb-4535-96bc-c30b51af5567\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:19:37.414078 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.413987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-z598v\" (UID: \"1d0d7097-31bb-4535-96bc-c30b51af5567\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:19:37.515606 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.515514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d0d7097-31bb-4535-96bc-c30b51af5567-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-z598v\" (UID: \"1d0d7097-31bb-4535-96bc-c30b51af5567\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:19:37.515606 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.515589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-z598v\" (UID: \"1d0d7097-31bb-4535-96bc-c30b51af5567\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:19:37.515834 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:37.515781    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 12:19:37.515878 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:37.515848    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert podName:1d0d7097-31bb-4535-96bc-c30b51af5567 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:38.015833439 +0000 UTC m=+51.543580474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-z598v" (UID: "1d0d7097-31bb-4535-96bc-c30b51af5567") : secret "networking-console-plugin-cert" not found
Jul 24 12:19:37.516364 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.516334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d0d7097-31bb-4535-96bc-c30b51af5567-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-z598v\" (UID: \"1d0d7097-31bb-4535-96bc-c30b51af5567\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:19:37.579324 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.579290    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:37.679839 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.679743    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:37.717496 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.717457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:37.717668 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:37.717629    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 12:19:37.717750 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:37.717717    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls podName:94cc8f44-c20d-472c-a11d-986174072f55 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:53.717694317 +0000 UTC m=+67.245441368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls") pod "dns-default-rxswx" (UID: "94cc8f44-c20d-472c-a11d-986174072f55") : secret "dns-default-metrics-tls" not found
Jul 24 12:19:37.818099 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:37.818066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:37.818290 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:37.818236    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 12:19:37.818290 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:37.818257    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847cc74b65-87hbw: secret "image-registry-tls" not found
Jul 24 12:19:37.818390 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:37.818327    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls podName:a1b2b5c6-4978-40e1-ad9a-a39876b680a7 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:53.818306581 +0000 UTC m=+67.346053638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls") pod "image-registry-847cc74b65-87hbw" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7") : secret "image-registry-tls" not found
Jul 24 12:19:38.020456 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:38.020417    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-z598v\" (UID: \"1d0d7097-31bb-4535-96bc-c30b51af5567\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:19:38.020714 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:38.020561    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 12:19:38.020714 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:38.020629    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert podName:1d0d7097-31bb-4535-96bc-c30b51af5567 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:39.020615296 +0000 UTC m=+52.548362328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-z598v" (UID: "1d0d7097-31bb-4535-96bc-c30b51af5567") : secret "networking-console-plugin-cert" not found
Jul 24 12:19:38.424683 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:38.424642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-d4jq6\" (UID: \"0d88bedb-dc43-4a7d-8a6c-375525c08823\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:19:38.425185 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:38.424764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:38.425185 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:38.424956    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 12:19:38.425185 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:38.425039    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls podName:4dcb8aa8-6eb1-4922-be33-3969dbe5ea51 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:46.425001353 +0000 UTC m=+59.952748405 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jlrvs" (UID: "4dcb8aa8-6eb1-4922-be33-3969dbe5ea51") : secret "cluster-monitoring-operator-tls" not found
Jul 24 12:19:38.425460 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:38.425433    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 12:19:38.425541 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:38.425513    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls podName:0d88bedb-dc43-4a7d-8a6c-375525c08823 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:46.425495188 +0000 UTC m=+59.953242224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls") pod "cluster-samples-operator-6748489656-d4jq6" (UID: "0d88bedb-dc43-4a7d-8a6c-375525c08823") : secret "samples-operator-tls" not found
Jul 24 12:19:38.526236 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:38.526150    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:38.526236 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:38.526234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:38.526483 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:38.526312    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 12:19:38.526483 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:38.526389    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle podName:7dcef893-f48b-41d9-acd1-50636e761528 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:46.526371411 +0000 UTC m=+60.054118458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle") pod "router-default-86b5bcffdf-j8mkb" (UID: "7dcef893-f48b-41d9-acd1-50636e761528") : configmap references non-existent config key: service-ca.crt
Jul 24 12:19:38.526483 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:38.526411    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs podName:7dcef893-f48b-41d9-acd1-50636e761528 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:46.52640192 +0000 UTC m=+60.054148955 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs") pod "router-default-86b5bcffdf-j8mkb" (UID: "7dcef893-f48b-41d9-acd1-50636e761528") : secret "router-metrics-certs-default" not found
Jul 24 12:19:39.031564 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.031533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-z598v\" (UID: \"1d0d7097-31bb-4535-96bc-c30b51af5567\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:19:39.031798 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:39.031734    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 12:19:39.031840 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:39.031812    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert podName:1d0d7097-31bb-4535-96bc-c30b51af5567 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:41.031792384 +0000 UTC m=+54.559539419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-z598v" (UID: "1d0d7097-31bb-4535-96bc-c30b51af5567") : secret "networking-console-plugin-cert" not found
Jul 24 12:19:39.223770 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.223739    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/0.log"
Jul 24 12:19:39.223945 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.223787    2566 generic.go:356] "Generic (PLEG): container finished" podID="e03b0875-5355-4282-96e5-efe11ccd7618" containerID="5ba3784e5282cdbabc72f1d4888502deb07dd26edb99a57b6fe83921ec20c123" exitCode=255
Jul 24 12:19:39.223945 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.223914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5" event={"ID":"e03b0875-5355-4282-96e5-efe11ccd7618","Type":"ContainerDied","Data":"5ba3784e5282cdbabc72f1d4888502deb07dd26edb99a57b6fe83921ec20c123"}
Jul 24 12:19:39.224144 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.224124    2566 scope.go:117] "RemoveContainer" containerID="5ba3784e5282cdbabc72f1d4888502deb07dd26edb99a57b6fe83921ec20c123"
Jul 24 12:19:39.227297 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.225701    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq" event={"ID":"0c42a44d-e70d-4baa-8fd9-5489b77c1e18","Type":"ContainerStarted","Data":"f642750186334f4b4bc82f3f1baab5bbce8d1cee45c30528a996d8684e0dfce6"}
Jul 24 12:19:39.228193 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.228171    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n" event={"ID":"f810f78d-7227-4bd0-aab4-194ced389280","Type":"ContainerStarted","Data":"4a01041906baa9df24dc2c681c1393b443534e78a3c3b1456c61370bfbff4163"}
Jul 24 12:19:39.229901 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.229879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zlm49" event={"ID":"45ae8a84-d47e-461b-bd9a-6d076a5b94f0","Type":"ContainerStarted","Data":"8527b115d5e4127ef156b209c6b53bb2ed66fc25b1aa279e727bbcbdfad1bc6c"}
Jul 24 12:19:39.231491 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.231470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4vl9k" event={"ID":"0d55faa5-7be5-4ff1-b37e-4c556baaa14e","Type":"ContainerStarted","Data":"64306f533552c74c45c192c75cae000f070c88b20cbe961280120bd9519f78d1"}
Jul 24 12:19:39.296851 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.296798    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-zlm49" podStartSLOduration=2.848412231 podStartE2EDuration="9.296777658s" totalImagesPullingTime="6.448365427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:39.295379722 +0000 UTC m=+52.823126797" watchObservedRunningTime="2026-07-24 12:19:39.296777658 +0000 UTC m=+52.824524714"
Jul 24 12:19:39.339512 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.338821    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq" podStartSLOduration=2.911236186 podStartE2EDuration="9.338800222s" totalImagesPullingTime="6.427564036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:39.338736978 +0000 UTC m=+52.866484033" watchObservedRunningTime="2026-07-24 12:19:39.338800222 +0000 UTC m=+52.866547277"
Jul 24 12:19:39.374191 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.373582    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:39.374191 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.373579    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n" podStartSLOduration=2.896857658 podStartE2EDuration="9.373558488s" totalImagesPullingTime="6.47670083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:39.371998577 +0000 UTC m=+52.899745633" watchObservedRunningTime="2026-07-24 12:19:39.373558488 +0000 UTC m=+52.901305545"
Jul 24 12:19:39.398793 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.398738    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4vl9k" podStartSLOduration=2.891131929 podStartE2EDuration="9.398718412s" totalImagesPullingTime="6.507586483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:39.398494785 +0000 UTC m=+52.926241841" watchObservedRunningTime="2026-07-24 12:19:39.398718412 +0000 UTC m=+52.926465469"
Jul 24 12:19:39.459886 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.459855    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:39.573711 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.573617    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:39.650270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.650239    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:39.907289 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.907218    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9swnm"]
Jul 24 12:19:39.922355 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.922325    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:39.924112 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.924080    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9swnm"]
Jul 24 12:19:39.927092 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.927072    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 12:19:39.928364 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.928341    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sd4k9\""
Jul 24 12:19:39.928610 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.928595    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 12:19:39.934064 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.934042    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:39.943055 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.943010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:39.943144 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.943063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:39.943144 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.943082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjfml\" (UniqueName: \"kubernetes.io/projected/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-api-access-tjfml\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:39.943228 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.943190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76b6e7bc-a18d-4155-aaa9-11a3a084be66-data-volume\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:39.943384 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:39.943343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76b6e7bc-a18d-4155-aaa9-11a3a084be66-crio-socket\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:40.016633 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.016596    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:40.043828 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.043799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76b6e7bc-a18d-4155-aaa9-11a3a084be66-crio-socket\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:40.043989 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.043868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:40.043989 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.043896    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:40.043989 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.043914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjfml\" (UniqueName: \"kubernetes.io/projected/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-api-access-tjfml\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:40.043989 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.043946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76b6e7bc-a18d-4155-aaa9-11a3a084be66-data-volume\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:40.044215 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:40.043999    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 12:19:40.044215 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.044006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76b6e7bc-a18d-4155-aaa9-11a3a084be66-crio-socket\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:40.044215 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:40.044072    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm podName:76b6e7bc-a18d-4155-aaa9-11a3a084be66 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:40.544055849 +0000 UTC m=+54.071802894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9swnm" (UID: "76b6e7bc-a18d-4155-aaa9-11a3a084be66") : configmap "kube-rbac-proxy" not found
Jul 24 12:19:40.044215 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:40.044073    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 12:19:40.044215 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:40.044126    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls podName:76b6e7bc-a18d-4155-aaa9-11a3a084be66 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:40.544109662 +0000 UTC m=+54.071856707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9swnm" (UID: "76b6e7bc-a18d-4155-aaa9-11a3a084be66") : secret "insights-runtime-extractor-tls" not found
Jul 24 12:19:40.044412 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.044376    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76b6e7bc-a18d-4155-aaa9-11a3a084be66-data-volume\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:40.055848 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.055824    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjfml\" (UniqueName: \"kubernetes.io/projected/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-api-access-tjfml\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:40.235805 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.235741    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:19:40.236145 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.236130    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/0.log"
Jul 24 12:19:40.236230 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.236164    2566 generic.go:356] "Generic (PLEG): container finished" podID="e03b0875-5355-4282-96e5-efe11ccd7618" containerID="7b00e80d735ec7581e777dd4763ed7961a3facae7fe053803ca5d02eb4623d38" exitCode=255
Jul 24 12:19:40.236297 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.236273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5" event={"ID":"e03b0875-5355-4282-96e5-efe11ccd7618","Type":"ContainerDied","Data":"7b00e80d735ec7581e777dd4763ed7961a3facae7fe053803ca5d02eb4623d38"}
Jul 24 12:19:40.236350 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.236331    2566 scope.go:117] "RemoveContainer" containerID="5ba3784e5282cdbabc72f1d4888502deb07dd26edb99a57b6fe83921ec20c123"
Jul 24 12:19:40.236564 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.236527    2566 scope.go:117] "RemoveContainer" containerID="7b00e80d735ec7581e777dd4763ed7961a3facae7fe053803ca5d02eb4623d38"
Jul 24 12:19:40.236742 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:40.236720    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-rfts5_openshift-console-operator(e03b0875-5355-4282-96e5-efe11ccd7618)\"" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5" podUID="e03b0875-5355-4282-96e5-efe11ccd7618"
Jul 24 12:19:40.279070 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.279036    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:40.369994 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.369960    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:40.549214 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.549180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:40.549604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:40.549220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:40.549604 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:40.549309    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 12:19:40.549604 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:40.549329    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 12:19:40.549604 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:40.549379    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm podName:76b6e7bc-a18d-4155-aaa9-11a3a084be66 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:41.549364919 +0000 UTC m=+55.077111951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9swnm" (UID: "76b6e7bc-a18d-4155-aaa9-11a3a084be66") : configmap "kube-rbac-proxy" not found
Jul 24 12:19:40.549604 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:40.549459    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls podName:76b6e7bc-a18d-4155-aaa9-11a3a084be66 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:41.549443474 +0000 UTC m=+55.077190507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9swnm" (UID: "76b6e7bc-a18d-4155-aaa9-11a3a084be66") : secret "insights-runtime-extractor-tls" not found
Jul 24 12:19:41.007935 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:41.007906    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:41.008109 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:41.007949    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:41.054618 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:41.054590    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-z598v\" (UID: \"1d0d7097-31bb-4535-96bc-c30b51af5567\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:19:41.054757 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:41.054738    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 12:19:41.054816 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:41.054807    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert podName:1d0d7097-31bb-4535-96bc-c30b51af5567 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:45.054792444 +0000 UTC m=+58.582539475 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-z598v" (UID: "1d0d7097-31bb-4535-96bc-c30b51af5567") : secret "networking-console-plugin-cert" not found
Jul 24 12:19:41.241106 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:41.241078    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:19:41.241422 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:41.241407    2566 scope.go:117] "RemoveContainer" containerID="7b00e80d735ec7581e777dd4763ed7961a3facae7fe053803ca5d02eb4623d38"
Jul 24 12:19:41.241580 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:41.241562    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-rfts5_openshift-console-operator(e03b0875-5355-4282-96e5-efe11ccd7618)\"" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5" podUID="e03b0875-5355-4282-96e5-efe11ccd7618"
Jul 24 12:19:41.559562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:41.559520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:41.559562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:41.559568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:41.560044 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:41.559664    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 12:19:41.560044 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:41.559731    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm podName:76b6e7bc-a18d-4155-aaa9-11a3a084be66 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:43.559715711 +0000 UTC m=+57.087462742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9swnm" (UID: "76b6e7bc-a18d-4155-aaa9-11a3a084be66") : configmap "kube-rbac-proxy" not found
Jul 24 12:19:41.560044 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:41.559756    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 12:19:41.560044 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:41.559808    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls podName:76b6e7bc-a18d-4155-aaa9-11a3a084be66 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:43.559794436 +0000 UTC m=+57.087541468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9swnm" (UID: "76b6e7bc-a18d-4155-aaa9-11a3a084be66") : secret "insights-runtime-extractor-tls" not found
Jul 24 12:19:41.924559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:41.924462    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:42.244558 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:42.244484    2566 scope.go:117] "RemoveContainer" containerID="7b00e80d735ec7581e777dd4763ed7961a3facae7fe053803ca5d02eb4623d38"
Jul 24 12:19:42.244693 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:42.244636    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-rfts5_openshift-console-operator(e03b0875-5355-4282-96e5-efe11ccd7618)\"" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5" podUID="e03b0875-5355-4282-96e5-efe11ccd7618"
Jul 24 12:19:42.270687 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:42.270667    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hjxk2_a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3/dns-node-resolver/0.log"
Jul 24 12:19:42.523997 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:42.523961    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:42.603728 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:42.603695    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:42.892776 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:42.892694    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:43.272948 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:43.272922    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n6d7j_f905cc3c-2a1f-4cfc-934d-e465550ec974/node-ca/0.log"
Jul 24 12:19:43.577775 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:43.577700    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:43.577775 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:43.577738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:43.577997 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:43.577826    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 12:19:43.577997 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:43.577919    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm podName:76b6e7bc-a18d-4155-aaa9-11a3a084be66 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:47.577902238 +0000 UTC m=+61.105649269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9swnm" (UID: "76b6e7bc-a18d-4155-aaa9-11a3a084be66") : configmap "kube-rbac-proxy" not found
Jul 24 12:19:43.577997 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:43.577934    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 12:19:43.577997 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:43.577992    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls podName:76b6e7bc-a18d-4155-aaa9-11a3a084be66 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:47.577975469 +0000 UTC m=+61.105722515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9swnm" (UID: "76b6e7bc-a18d-4155-aaa9-11a3a084be66") : secret "insights-runtime-extractor-tls" not found
Jul 24 12:19:43.989655 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:43.989570    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:44.070229 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:44.070205    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nfjm7_d6224e74-72f6-4d5c-bbbd-d87bb40d9507/serve-healthcheck-canary/0.log"
Jul 24 12:19:44.089935 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:44.089897    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:44.670260 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:44.670222    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n_f810f78d-7227-4bd0-aab4-194ced389280/kube-storage-version-migrator-operator/0.log"
Jul 24 12:19:45.090936 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:45.090897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-z598v\" (UID: \"1d0d7097-31bb-4535-96bc-c30b51af5567\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:19:45.091445 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:45.091103    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 12:19:45.091445 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:45.091181    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert podName:1d0d7097-31bb-4535-96bc-c30b51af5567 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:53.091162077 +0000 UTC m=+66.618909316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-z598v" (UID: "1d0d7097-31bb-4535-96bc-c30b51af5567") : secret "networking-console-plugin-cert" not found
Jul 24 12:19:46.359792 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:46.359761    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:46.501009 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:46.500971    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-d4jq6\" (UID: \"0d88bedb-dc43-4a7d-8a6c-375525c08823\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:19:46.501216 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:46.501096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:19:46.501216 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:46.501130    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 12:19:46.501216 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:46.501209    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls podName:0d88bedb-dc43-4a7d-8a6c-375525c08823 nodeName:}" failed. No retries permitted until 2026-07-24 12:20:02.5011879 +0000 UTC m=+76.028934960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls") pod "cluster-samples-operator-6748489656-d4jq6" (UID: "0d88bedb-dc43-4a7d-8a6c-375525c08823") : secret "samples-operator-tls" not found
Jul 24 12:19:46.501344 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:46.501219    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 12:19:46.501344 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:46.501254    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls podName:4dcb8aa8-6eb1-4922-be33-3969dbe5ea51 nodeName:}" failed. No retries permitted until 2026-07-24 12:20:02.501244243 +0000 UTC m=+76.028991275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jlrvs" (UID: "4dcb8aa8-6eb1-4922-be33-3969dbe5ea51") : secret "cluster-monitoring-operator-tls" not found
Jul 24 12:19:46.601602 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:46.601570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:46.601729 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:46.601629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:19:46.601766 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:46.601740    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 12:19:46.601808 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:46.601744    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle podName:7dcef893-f48b-41d9-acd1-50636e761528 nodeName:}" failed. No retries permitted until 2026-07-24 12:20:02.601729891 +0000 UTC m=+76.129476922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle") pod "router-default-86b5bcffdf-j8mkb" (UID: "7dcef893-f48b-41d9-acd1-50636e761528") : configmap references non-existent config key: service-ca.crt
Jul 24 12:19:46.601853 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:46.601808    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs podName:7dcef893-f48b-41d9-acd1-50636e761528 nodeName:}" failed. No retries permitted until 2026-07-24 12:20:02.601791097 +0000 UTC m=+76.129538137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs") pod "router-default-86b5bcffdf-j8mkb" (UID: "7dcef893-f48b-41d9-acd1-50636e761528") : secret "router-metrics-certs-default" not found
Jul 24 12:19:46.898866 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:46.898826    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:47.023727 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:47.023651    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:47.128950 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:47.128910    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:47.301914 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:47.301877    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:47.400112 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:47.400084    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:47.610707 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:47.610616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:47.610707 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:47.610661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:47.610940 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:47.610749    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 12:19:47.610940 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:47.610772    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 12:19:47.610940 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:47.610835    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm podName:76b6e7bc-a18d-4155-aaa9-11a3a084be66 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:55.610819738 +0000 UTC m=+69.138566770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9swnm" (UID: "76b6e7bc-a18d-4155-aaa9-11a3a084be66") : configmap "kube-rbac-proxy" not found
Jul 24 12:19:47.610940 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:47.610850    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls podName:76b6e7bc-a18d-4155-aaa9-11a3a084be66 nodeName:}" failed. No retries permitted until 2026-07-24 12:19:55.610843198 +0000 UTC m=+69.138590231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9swnm" (UID: "76b6e7bc-a18d-4155-aaa9-11a3a084be66") : secret "insights-runtime-extractor-tls" not found
Jul 24 12:19:47.618293 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:47.618267    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:48.123310 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:48.123282    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ccs4"
Jul 24 12:19:48.389928 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:48.389836    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:48.605049 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:48.604982    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:49.012414 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:49.012380    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:50.043434 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:50.043396    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:50.144432 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:50.144406    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:50.393314 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:50.393243    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:50.624520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:50.624485    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:50.799488 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:50.799451    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:51.021504 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:51.021472    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:51.192558 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:51.192474    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:51.594133 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:51.594103    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:51.651486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:51.651459    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:51.990832 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:51.990744    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:52.200555 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.200524    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:52.653331 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.653296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:52.653525 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.653366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:19:52.656520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.656503    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 12:19:52.656574 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.656547    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 12:19:52.664096 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:52.664077    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 12:19:52.664170 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:52.664149    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs podName:7585202d-e3a1-46b8-90e8-316041c7e09f nodeName:}" failed. No retries permitted until 2026-07-24 12:20:56.66412721 +0000 UTC m=+130.191874242 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs") pod "network-metrics-daemon-zdv26" (UID: "7585202d-e3a1-46b8-90e8-316041c7e09f") : secret "metrics-daemon-secret" not found
Jul 24 12:19:52.666411 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.666383    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17a4fb33-440c-4563-b7e1-b9ee4ef34d90-original-pull-secret\") pod \"global-pull-secret-syncer-gnrrg\" (UID: \"17a4fb33-440c-4563-b7e1-b9ee4ef34d90\") " pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:52.698648 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.698621    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gnrrg"
Jul 24 12:19:52.795462 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.795418    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:52.821350 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.821281    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gnrrg"]
Jul 24 12:19:52.823490 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:19:52.823465    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a4fb33_440c_4563_b7e1_b9ee4ef34d90.slice/crio-7af03c44e89777f68e3e79723752ccb2b64e9a049c73cc64462302fbcf53adb2 WatchSource:0}: Error finding container 7af03c44e89777f68e3e79723752ccb2b64e9a049c73cc64462302fbcf53adb2: Status 404 returned error can't find the container with id 7af03c44e89777f68e3e79723752ccb2b64e9a049c73cc64462302fbcf53adb2
Jul 24 12:19:52.854653 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.854623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnt4\" (UniqueName: \"kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4\") pod \"network-check-target-xmjh2\" (UID: \"549ca555-24a4-4899-9071-f2776523a390\") " pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:52.857075 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.857058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnt4\" (UniqueName: \"kubernetes.io/projected/549ca555-24a4-4899-9071-f2776523a390-kube-api-access-xgnt4\") pod \"network-check-target-xmjh2\" (UID: \"549ca555-24a4-4899-9071-f2776523a390\") " pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:52.990590 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.990515    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k4xp5\""
Jul 24 12:19:52.998261 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:52.998229    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:53.082396 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.082354    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:53.127052 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.127012    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xmjh2"]
Jul 24 12:19:53.129543 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:19:53.129513    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549ca555_24a4_4899_9071_f2776523a390.slice/crio-3440ea67222538d6ee5c5535975a2549bc041e537332f27a2e802344222fe62f WatchSource:0}: Error finding container 3440ea67222538d6ee5c5535975a2549bc041e537332f27a2e802344222fe62f: Status 404 returned error can't find the container with id 3440ea67222538d6ee5c5535975a2549bc041e537332f27a2e802344222fe62f
Jul 24 12:19:53.158965 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.158942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-z598v\" (UID: \"1d0d7097-31bb-4535-96bc-c30b51af5567\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:19:53.159135 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:53.159117    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 12:19:53.159202 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:53.159195    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert podName:1d0d7097-31bb-4535-96bc-c30b51af5567 nodeName:}" failed. No retries permitted until 2026-07-24 12:20:09.159172777 +0000 UTC m=+82.686919812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-z598v" (UID: "1d0d7097-31bb-4535-96bc-c30b51af5567") : secret "networking-console-plugin-cert" not found
Jul 24 12:19:53.277680 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.277650    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xmjh2" event={"ID":"549ca555-24a4-4899-9071-f2776523a390","Type":"ContainerStarted","Data":"dd07cf6c83cdf36296ddb3440fbbc1514f03fa1b1c153ceb6bf2f04ac7bc532c"}
Jul 24 12:19:53.278133 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.277696    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xmjh2" event={"ID":"549ca555-24a4-4899-9071-f2776523a390","Type":"ContainerStarted","Data":"3440ea67222538d6ee5c5535975a2549bc041e537332f27a2e802344222fe62f"}
Jul 24 12:19:53.278133 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.277745    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:19:53.278818 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.278800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gnrrg" event={"ID":"17a4fb33-440c-4563-b7e1-b9ee4ef34d90","Type":"ContainerStarted","Data":"7af03c44e89777f68e3e79723752ccb2b64e9a049c73cc64462302fbcf53adb2"}
Jul 24 12:19:53.295180 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.295133    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xmjh2" podStartSLOduration=66.295119258 podStartE2EDuration="1m6.295119258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:53.294150399 +0000 UTC m=+66.821897454" watchObservedRunningTime="2026-07-24 12:19:53.295119258 +0000 UTC m=+66.822866312"
Jul 24 12:19:53.409311 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.409276    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:53.557248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.557138    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:53.658719 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.658634    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:53.765945 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.765908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:53.768586 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.768562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94cc8f44-c20d-472c-a11d-986174072f55-metrics-tls\") pod \"dns-default-rxswx\" (UID: \"94cc8f44-c20d-472c-a11d-986174072f55\") " pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:53.800220 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.800172    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:53.867390 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.867304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:53.870093 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.870063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls\") pod \"image-registry-847cc74b65-87hbw\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") " pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:53.948757 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.948728    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xlzt9\""
Jul 24 12:19:53.956283 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.956254    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:53.968950 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:53.968895    2566 scope.go:117] "RemoveContainer" containerID="7b00e80d735ec7581e777dd4763ed7961a3facae7fe053803ca5d02eb4623d38"
Jul 24 12:19:54.013044 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.012987    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-68vm8\""
Jul 24 12:19:54.020973 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.020944    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:54.126180 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.125758    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-847cc74b65-87hbw"]
Jul 24 12:19:54.129155 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:19:54.129116    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b2b5c6_4978_40e1_ad9a_a39876b680a7.slice/crio-03a8e74bfd7e5486b84c9cf240a657a5f0348707db1178f546e774c98e8cc21f WatchSource:0}: Error finding container 03a8e74bfd7e5486b84c9cf240a657a5f0348707db1178f546e774c98e8cc21f: Status 404 returned error can't find the container with id 03a8e74bfd7e5486b84c9cf240a657a5f0348707db1178f546e774c98e8cc21f
Jul 24 12:19:54.184209 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.184182    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rxswx"]
Jul 24 12:19:54.187143 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:19:54.187106    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94cc8f44_c20d_472c_a11d_986174072f55.slice/crio-b4a3432856e3c99b7abb3ad15b63f32f0495109666c1c6e2a03a94fa7ae28eae WatchSource:0}: Error finding container b4a3432856e3c99b7abb3ad15b63f32f0495109666c1c6e2a03a94fa7ae28eae: Status 404 returned error can't find the container with id b4a3432856e3c99b7abb3ad15b63f32f0495109666c1c6e2a03a94fa7ae28eae
Jul 24 12:19:54.193470 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.193445    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:54.284234 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.284156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rxswx" event={"ID":"94cc8f44-c20d-472c-a11d-986174072f55","Type":"ContainerStarted","Data":"b4a3432856e3c99b7abb3ad15b63f32f0495109666c1c6e2a03a94fa7ae28eae"}
Jul 24 12:19:54.286687 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.286660    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:19:54.286862 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.286762    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5" event={"ID":"e03b0875-5355-4282-96e5-efe11ccd7618","Type":"ContainerStarted","Data":"0dba8890a8c4b70dbf23f7a54ce73fd7e8e9b6d519e86b26655ca4824be5fda4"}
Jul 24 12:19:54.288570 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.287163    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:54.288866 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.288734    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-847cc74b65-87hbw" event={"ID":"a1b2b5c6-4978-40e1-ad9a-a39876b680a7","Type":"ContainerStarted","Data":"8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e"}
Jul 24 12:19:54.288866 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.288779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-847cc74b65-87hbw" event={"ID":"a1b2b5c6-4978-40e1-ad9a-a39876b680a7","Type":"ContainerStarted","Data":"03a8e74bfd7e5486b84c9cf240a657a5f0348707db1178f546e774c98e8cc21f"}
Jul 24 12:19:54.288866 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.288818    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:19:54.307515 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.307472    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5" podStartSLOduration=17.81384416 podStartE2EDuration="24.307457152s" totalImagesPullingTime="6.493612992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:54.305902791 +0000 UTC m=+67.833649846" watchObservedRunningTime="2026-07-24 12:19:54.307457152 +0000 UTC m=+67.835204202"
Jul 24 12:19:54.325317 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.325265    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-847cc74b65-87hbw" podStartSLOduration=67.325245762 podStartE2EDuration="1m7.325245762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:54.325119479 +0000 UTC m=+67.852866533" watchObservedRunningTime="2026-07-24 12:19:54.325245762 +0000 UTC m=+67.852992817"
Jul 24 12:19:54.608588 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.608552    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:54.791971 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:54.791931    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:55.198856 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:55.198801    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:55.287411 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:55.287371    2566 patch_prober.go:28] interesting pod/console-operator-584dcb6b54-rfts5 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": context deadline exceeded" start-of-body=
Jul 24 12:19:55.287866 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:55.287450    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5" podUID="e03b0875-5355-4282-96e5-efe11ccd7618" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": context deadline exceeded"
Jul 24 12:19:55.598559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:55.598529    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:55.691869 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:55.691834    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:55.692092 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:55.691883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:55.692169 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:55.692089    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 12:19:55.692169 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:19:55.692144    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls podName:76b6e7bc-a18d-4155-aaa9-11a3a084be66 nodeName:}" failed. No retries permitted until 2026-07-24 12:20:11.692128878 +0000 UTC m=+85.219875923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9swnm" (UID: "76b6e7bc-a18d-4155-aaa9-11a3a084be66") : secret "insights-runtime-extractor-tls" not found
Jul 24 12:19:55.693042 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:55.692997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b6e7bc-a18d-4155-aaa9-11a3a084be66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:19:56.000995 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:56.000941    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:56.201840 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:56.201812    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-rfts5"
Jul 24 12:19:56.382480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:56.382392    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:56.483289 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:56.483254    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:56.561700 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:56.561658    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:56.755034 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:56.754988    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:56.853655 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:56.853609    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:56.957492 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:56.957456    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:57.054540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:57.054390    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:57.189226 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:57.189194    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:57.290367 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:57.290272    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:57.347420 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:57.347342    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:57.596180 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:57.596152    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:57.695944 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:57.695863    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:57.795766 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:57.795737    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:57.896887 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:57.896801    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:58.070128 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:58.070091    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:58.305750 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:58.305719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gnrrg" event={"ID":"17a4fb33-440c-4563-b7e1-b9ee4ef34d90","Type":"ContainerStarted","Data":"dce9874f21e38a8f7e6092b0b115c48bd35a8ea3890fac9ccac2d65e67d012ed"}
Jul 24 12:19:58.307603 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:58.307577    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rxswx" event={"ID":"94cc8f44-c20d-472c-a11d-986174072f55","Type":"ContainerStarted","Data":"bcafcb39f0adce3ed7fc6dde4561012007969b51fe03671ffbcf3005522a6a94"}
Jul 24 12:19:58.307728 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:58.307625    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rxswx" event={"ID":"94cc8f44-c20d-472c-a11d-986174072f55","Type":"ContainerStarted","Data":"8d21ad4c0d1feed67412215a913194715cf68d7ae818059666b462e1528bf7dd"}
Jul 24 12:19:58.307728 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:58.307646    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rxswx"
Jul 24 12:19:58.322224 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:58.322184    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gnrrg" podStartSLOduration=66.559848051 podStartE2EDuration="1m11.322171741s" totalImagesPullingTime="4.76232369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:58.321447081 +0000 UTC m=+71.849194135" watchObservedRunningTime="2026-07-24 12:19:58.322171741 +0000 UTC m=+71.849918794"
Jul 24 12:19:58.342791 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:58.342752    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rxswx" podStartSLOduration=33.948369861 podStartE2EDuration="37.342739037s" totalImagesPullingTime="3.394369176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:58.342317415 +0000 UTC m=+71.870064468" watchObservedRunningTime="2026-07-24 12:19:58.342739037 +0000 UTC m=+71.870486072"
Jul 24 12:19:58.471816 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:58.471771    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:58.660558 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:58.660477    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:58.883166 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:58.883133    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:19:58.942937 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:19:58.942868    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:02.557496 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.557451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:20:02.557876 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.557587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-d4jq6\" (UID: \"0d88bedb-dc43-4a7d-8a6c-375525c08823\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:20:02.560045 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.559998    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d88bedb-dc43-4a7d-8a6c-375525c08823-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-d4jq6\" (UID: \"0d88bedb-dc43-4a7d-8a6c-375525c08823\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:20:02.560160 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.560081    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dcb8aa8-6eb1-4922-be33-3969dbe5ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jlrvs\" (UID: \"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:20:02.658756 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.658717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:20:02.658756 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.658757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:20:02.659338 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.659323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dcef893-f48b-41d9-acd1-50636e761528-service-ca-bundle\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:20:02.661160 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.661144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcef893-f48b-41d9-acd1-50636e761528-metrics-certs\") pod \"router-default-86b5bcffdf-j8mkb\" (UID: \"7dcef893-f48b-41d9-acd1-50636e761528\") " pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:20:02.687573 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.687544    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xvs99\""
Jul 24 12:20:02.695187 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.695171    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"
Jul 24 12:20:02.729288 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.729264    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-c287l\""
Jul 24 12:20:02.736946 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.736925    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"
Jul 24 12:20:02.829388 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.829358    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4lmzg\""
Jul 24 12:20:02.837717 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.837637    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:20:02.882879 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.882860    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6"]
Jul 24 12:20:02.906582 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.906558    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs"]
Jul 24 12:20:02.910434 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:20:02.910204    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcb8aa8_6eb1_4922_be33_3969dbe5ea51.slice/crio-70d44ff8772e8bc8e77ffbe2e39c74694bac210fc8a335f650f3aa46d724a5bf WatchSource:0}: Error finding container 70d44ff8772e8bc8e77ffbe2e39c74694bac210fc8a335f650f3aa46d724a5bf: Status 404 returned error can't find the container with id 70d44ff8772e8bc8e77ffbe2e39c74694bac210fc8a335f650f3aa46d724a5bf
Jul 24 12:20:02.995039 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:02.995000    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86b5bcffdf-j8mkb"]
Jul 24 12:20:02.997175 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:20:02.997150    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dcef893_f48b_41d9_acd1_50636e761528.slice/crio-b7c31fd4ff680f1c92ce99bf43349a5dcad04dc77bbce0940617a4214c975913 WatchSource:0}: Error finding container b7c31fd4ff680f1c92ce99bf43349a5dcad04dc77bbce0940617a4214c975913: Status 404 returned error can't find the container with id b7c31fd4ff680f1c92ce99bf43349a5dcad04dc77bbce0940617a4214c975913
Jul 24 12:20:03.328114 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:03.328006    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6" event={"ID":"0d88bedb-dc43-4a7d-8a6c-375525c08823","Type":"ContainerStarted","Data":"ec2fb037f3b664441ca49bb70cff18099b536e7c8c81320841ad1521122b6091"}
Jul 24 12:20:03.329720 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:03.329682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86b5bcffdf-j8mkb" event={"ID":"7dcef893-f48b-41d9-acd1-50636e761528","Type":"ContainerStarted","Data":"bd19785f75f2668b6ca38902127d03baffe39331ab32a69e8d67dc4e8f095cba"}
Jul 24 12:20:03.329879 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:03.329751    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86b5bcffdf-j8mkb" event={"ID":"7dcef893-f48b-41d9-acd1-50636e761528","Type":"ContainerStarted","Data":"b7c31fd4ff680f1c92ce99bf43349a5dcad04dc77bbce0940617a4214c975913"}
Jul 24 12:20:03.330854 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:03.330824    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs" event={"ID":"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51","Type":"ContainerStarted","Data":"70d44ff8772e8bc8e77ffbe2e39c74694bac210fc8a335f650f3aa46d724a5bf"}
Jul 24 12:20:03.352493 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:03.352414    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-86b5bcffdf-j8mkb" podStartSLOduration=33.352399924 podStartE2EDuration="33.352399924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:03.350149291 +0000 UTC m=+76.877896389" watchObservedRunningTime="2026-07-24 12:20:03.352399924 +0000 UTC m=+76.880146978"
Jul 24 12:20:03.838725 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:03.838683    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:20:03.841854 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:03.841830    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:20:03.962897 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:03.962855    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:04.061044 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:04.060981    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:04.160941 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:04.160823    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:04.226076 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:04.226039    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:04.336273 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:04.336245    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:20:04.337449 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:04.337426    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-86b5bcffdf-j8mkb"
Jul 24 12:20:05.095168 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:05.095129    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:05.155381 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:05.155293    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:05.518395 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:05.518365    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:05.573766 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:05.573733    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:06.092282 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:06.092247    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:06.195168 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:06.195133    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:06.256767 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:06.256726    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:06.341003 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:06.340971    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:06.344524 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:06.344447    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs" event={"ID":"4dcb8aa8-6eb1-4922-be33-3969dbe5ea51","Type":"ContainerStarted","Data":"1d71ec36e16b0bff57c5f2762c9c7f4ab196377b98958574aa90b32cd525aaa9"}
Jul 24 12:20:06.346350 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:06.346313    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6" event={"ID":"0d88bedb-dc43-4a7d-8a6c-375525c08823","Type":"ContainerStarted","Data":"428a65efef76e3c14d478e17358ce3ca6e2b6772c6b3876f10b13ef885d6cc29"}
Jul 24 12:20:06.346440 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:06.346365    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6" event={"ID":"0d88bedb-dc43-4a7d-8a6c-375525c08823","Type":"ContainerStarted","Data":"892c5ed6362454a97b903d4eba55c55154a3e3a9b1ee909c61d7886226e22a27"}
Jul 24 12:20:06.363010 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:06.362961    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jlrvs" podStartSLOduration=33.620021919 podStartE2EDuration="36.362944527s" totalImagesPullingTime="2.742922608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:06.362917443 +0000 UTC m=+79.890664497" watchObservedRunningTime="2026-07-24 12:20:06.362944527 +0000 UTC m=+79.890691580"
Jul 24 12:20:07.008163 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:07.008134    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:07.109638 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:07.109598    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:07.681299 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:07.681265    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:07.742145 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:07.742115    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:08.313900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:08.313870    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rxswx"
Jul 24 12:20:08.334113 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:08.334066    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-d4jq6" podStartSLOduration=35.643695593 podStartE2EDuration="38.33404998s" totalImagesPullingTime="2.690354387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:06.382940069 +0000 UTC m=+79.910687122" watchObservedRunningTime="2026-07-24 12:20:08.33404998 +0000 UTC m=+81.861797033"
Jul 24 12:20:08.587310 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:08.587228    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:08.652422 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:08.652393    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:09.224727 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:09.224690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-z598v\" (UID: \"1d0d7097-31bb-4535-96bc-c30b51af5567\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:20:09.227226 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:09.227192    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:09.227528 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:09.227509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d0d7097-31bb-4535-96bc-c30b51af5567-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-z598v\" (UID: \"1d0d7097-31bb-4535-96bc-c30b51af5567\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:20:09.327187 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:09.327148    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:09.508751 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:09.508725    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tz7fl\""
Jul 24 12:20:09.516368 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:09.516348    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v"
Jul 24 12:20:09.584779 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:09.584742    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:09.673996 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:09.673956    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-z598v"]
Jul 24 12:20:09.675961 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:20:09.675935    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0d7097_31bb_4535_96bc_c30b51af5567.slice/crio-ca7f6d09078630dfebf11c38b0f3f10484a847eec9e636d636f5a8aa10b237d4 WatchSource:0}: Error finding container ca7f6d09078630dfebf11c38b0f3f10484a847eec9e636d636f5a8aa10b237d4: Status 404 returned error can't find the container with id ca7f6d09078630dfebf11c38b0f3f10484a847eec9e636d636f5a8aa10b237d4
Jul 24 12:20:10.362868 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:10.362826    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v" event={"ID":"1d0d7097-31bb-4535-96bc-c30b51af5567","Type":"ContainerStarted","Data":"ca7f6d09078630dfebf11c38b0f3f10484a847eec9e636d636f5a8aa10b237d4"}
Jul 24 12:20:11.748155 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:11.748129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:20:11.750554 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:11.750535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b6e7bc-a18d-4155-aaa9-11a3a084be66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9swnm\" (UID: \"76b6e7bc-a18d-4155-aaa9-11a3a084be66\") " pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:20:12.037602 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:12.037527    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sd4k9\""
Jul 24 12:20:12.044087 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:12.044067    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9swnm"
Jul 24 12:20:12.205851 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:12.205830    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9swnm"]
Jul 24 12:20:12.208402 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:20:12.208372    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b6e7bc_a18d_4155_aaa9_11a3a084be66.slice/crio-a79145cbd28c9ccb4673a9c21775daee56875f07ef1249355ba4f39f467c3106 WatchSource:0}: Error finding container a79145cbd28c9ccb4673a9c21775daee56875f07ef1249355ba4f39f467c3106: Status 404 returned error can't find the container with id a79145cbd28c9ccb4673a9c21775daee56875f07ef1249355ba4f39f467c3106
Jul 24 12:20:12.374373 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:12.374272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9swnm" event={"ID":"76b6e7bc-a18d-4155-aaa9-11a3a084be66","Type":"ContainerStarted","Data":"a79145cbd28c9ccb4673a9c21775daee56875f07ef1249355ba4f39f467c3106"}
Jul 24 12:20:12.375959 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:12.375932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v" event={"ID":"1d0d7097-31bb-4535-96bc-c30b51af5567","Type":"ContainerStarted","Data":"de02c83dea8325e8752c845e8acea87090f8dc8dd4ecbabbc3b91a895d21e502"}
Jul 24 12:20:12.391815 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:12.391772    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-z598v" podStartSLOduration=33.746966321 podStartE2EDuration="35.391757478s" totalImagesPullingTime="1.644791157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:12.391294697 +0000 UTC m=+85.919041746" watchObservedRunningTime="2026-07-24 12:20:12.391757478 +0000 UTC m=+85.919504531"
Jul 24 12:20:12.519328 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:12.519287    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:12.619834 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:12.619804    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:12.736486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:12.736390    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:12.836524 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:12.836481    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:13.886747 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:13.886705    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:13.948520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:13.948484    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:13.962165 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:13.962129    2566 patch_prober.go:28] interesting pod/image-registry-847cc74b65-87hbw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 24 12:20:13.962288 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:13.962189    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-847cc74b65-87hbw" podUID="a1b2b5c6-4978-40e1-ad9a-a39876b680a7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 12:20:15.298648 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:15.298621    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:20:15.539110 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:15.539075    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:15.640529 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:15.640398    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:16.704721 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:16.704689    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:17.302138 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:17.302042    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:17.397710 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:17.397676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9swnm" event={"ID":"76b6e7bc-a18d-4155-aaa9-11a3a084be66","Type":"ContainerStarted","Data":"63d57e155a2fd2719d61d9bef5af997acc05e1d0f5884a30655025193a50f32b"}
Jul 24 12:20:17.404675 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:17.404617    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:17.503210 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:17.503163    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:18.371388 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.371336    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:18.596949 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.596916    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:18.626423 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.626350    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xs7tj"]
Jul 24 12:20:18.631428 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.631396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.636078 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.635802    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 12:20:18.636078 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.635842    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 12:20:18.636259 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.636112    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wc5gs\""
Jul 24 12:20:18.636259 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.636219    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 12:20:18.636369 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.636335    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 12:20:18.674374 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.674330    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:18.705400 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.705365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9072e97-484e-4720-b8b7-8339a3ad2d85-metrics-client-ca\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.705583 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.705467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.705583 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.705510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9072e97-484e-4720-b8b7-8339a3ad2d85-sys\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.705583 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.705541    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c9072e97-484e-4720-b8b7-8339a3ad2d85-root\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.705696 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.705605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9h75\" (UniqueName: \"kubernetes.io/projected/c9072e97-484e-4720-b8b7-8339a3ad2d85-kube-api-access-w9h75\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.705696 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.705646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-wtmp\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.705773 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.705713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-textfile\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.705773 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.705743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-tls\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.705860 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.705772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-accelerators-collector-config\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.760946 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.760822    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:18.806777 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.806745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-textfile\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.806777 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.806782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-tls\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807040 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.806806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-accelerators-collector-config\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807040 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.806856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9072e97-484e-4720-b8b7-8339a3ad2d85-metrics-client-ca\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807040 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.806906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807040 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.806939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9072e97-484e-4720-b8b7-8339a3ad2d85-sys\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807040 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.806959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c9072e97-484e-4720-b8b7-8339a3ad2d85-root\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807040 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.806997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9h75\" (UniqueName: \"kubernetes.io/projected/c9072e97-484e-4720-b8b7-8339a3ad2d85-kube-api-access-w9h75\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807040 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.807040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-wtmp\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807348 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.807085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9072e97-484e-4720-b8b7-8339a3ad2d85-sys\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807348 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.807153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c9072e97-484e-4720-b8b7-8339a3ad2d85-root\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807348 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.807225    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-wtmp\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.807463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-textfile\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807929 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.807881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9072e97-484e-4720-b8b7-8339a3ad2d85-metrics-client-ca\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.807929 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.807925    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-accelerators-collector-config\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.810419 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.810383    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.810515 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.810484    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c9072e97-484e-4720-b8b7-8339a3ad2d85-node-exporter-tls\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.815919 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.815899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9h75\" (UniqueName: \"kubernetes.io/projected/c9072e97-484e-4720-b8b7-8339a3ad2d85-kube-api-access-w9h75\") pod \"node-exporter-xs7tj\" (UID: \"c9072e97-484e-4720-b8b7-8339a3ad2d85\") " pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.884532 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.884413    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:18.943478 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.943446    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xs7tj"
Jul 24 12:20:18.983484 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:18.983450    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:19.083498 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:19.083458    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:19.184331 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:19.183838    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:19.283755 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:19.283719    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:19.596807 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:19.596765    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:19.697704 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:19.697672    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:19.931134 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:19.931057    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:20.515125 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:20:20.515101    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9072e97_484e_4720_b8b7_8339a3ad2d85.slice/crio-7c4019cb3a7453c4002a34d85dd6982edc71d6c7ab10619da5c1eececbaa5764 WatchSource:0}: Error finding container 7c4019cb3a7453c4002a34d85dd6982edc71d6c7ab10619da5c1eececbaa5764: Status 404 returned error can't find the container with id 7c4019cb3a7453c4002a34d85dd6982edc71d6c7ab10619da5c1eececbaa5764
Jul 24 12:20:21.418059 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:21.418003    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9swnm" event={"ID":"76b6e7bc-a18d-4155-aaa9-11a3a084be66","Type":"ContainerStarted","Data":"b930ef46e76211e0ae5f2606c851682c42999a66524de61592f9e3bde8ed2bf6"}
Jul 24 12:20:21.419315 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:21.419276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xs7tj" event={"ID":"c9072e97-484e-4720-b8b7-8339a3ad2d85","Type":"ContainerStarted","Data":"7c4019cb3a7453c4002a34d85dd6982edc71d6c7ab10619da5c1eececbaa5764"}
Jul 24 12:20:21.584603 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:21.584572    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:22.004988 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:22.004946    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:22.059043 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:22.058983    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:22.425764 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:22.425693    2566 generic.go:356] "Generic (PLEG): container finished" podID="c9072e97-484e-4720-b8b7-8339a3ad2d85" containerID="23c1bff8b8909db7d388212bf261df9b0998a47020ab45ee95d49c18f452fdd0" exitCode=0
Jul 24 12:20:22.426206 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:22.425775    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xs7tj" event={"ID":"c9072e97-484e-4720-b8b7-8339a3ad2d85","Type":"ContainerDied","Data":"23c1bff8b8909db7d388212bf261df9b0998a47020ab45ee95d49c18f452fdd0"}
Jul 24 12:20:23.416122 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:23.416078    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:23.533380 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:23.533338    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:24.291676 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:24.291645    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xmjh2"
Jul 24 12:20:24.350229 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:24.350194    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:24.436617 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:24.436533    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9swnm" event={"ID":"76b6e7bc-a18d-4155-aaa9-11a3a084be66","Type":"ContainerStarted","Data":"d86719aec54acd4f8f3f3e75d2e02fce67a31083d914091cc24be6c28d1869d6"}
Jul 24 12:20:24.438625 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:24.438596    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xs7tj" event={"ID":"c9072e97-484e-4720-b8b7-8339a3ad2d85","Type":"ContainerStarted","Data":"dddbe3f55027726d3d19a526c42c7eecb57cd45a27746a6a762e70184c7e2892"}
Jul 24 12:20:24.438767 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:24.438642    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xs7tj" event={"ID":"c9072e97-484e-4720-b8b7-8339a3ad2d85","Type":"ContainerStarted","Data":"c886b31bf83a8c792183f3f12c5b8053159b72ac0e52da173bf2ce450c9504ec"}
Jul 24 12:20:24.459482 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:24.459440    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9swnm" podStartSLOduration=33.628414384 podStartE2EDuration="45.459426643s" totalImagesPullingTime="11.831012259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:19:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:24.45822354 +0000 UTC m=+97.985970596" watchObservedRunningTime="2026-07-24 12:20:24.459426643 +0000 UTC m=+97.987173694"
Jul 24 12:20:24.480401 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:24.480358    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xs7tj" podStartSLOduration=5.46508653 podStartE2EDuration="6.480345619s" totalImagesPullingTime="931.88566ms" totalInitContainerRuntime="83.373429ms" isStatefulPod=true podCreationTimestamp="2026-07-24 12:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:24.479799083 +0000 UTC m=+98.007546137" watchObservedRunningTime="2026-07-24 12:20:24.480345619 +0000 UTC m=+98.008092672"
Jul 24 12:20:24.838848 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:24.838808    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:25.535571 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:25.535538    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-847cc74b65-87hbw"]
Jul 24 12:20:25.619709 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:25.619675    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:25.720188 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:25.720154    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:26.901190 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:26.901097    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:27.007935 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:27.007895    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:28.950139 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:28.950102    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:29.051549 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:29.051511    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:29.115588 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:29.115545    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:29.174404 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:29.174370    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:34.252291 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:34.252228    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:34.352414 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:34.352374    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:35.116541 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:35.116505    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:35.202073 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:35.202046    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:36.022693 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:36.022658    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:36.110134 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:36.110099    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:36.200006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:36.199971    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:36.254739 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:36.254704    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:38.193499 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:38.193461    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:38.293302 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:38.293270    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:45.285644 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:45.285581    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:45.346391 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:45.346363    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:45.398860 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:45.398827    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:49.530663 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:49.530626    2566 generic.go:356] "Generic (PLEG): container finished" podID="f810f78d-7227-4bd0-aab4-194ced389280" containerID="4a01041906baa9df24dc2c681c1393b443534e78a3c3b1456c61370bfbff4163" exitCode=0
Jul 24 12:20:49.531051 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:49.530702    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n" event={"ID":"f810f78d-7227-4bd0-aab4-194ced389280","Type":"ContainerDied","Data":"4a01041906baa9df24dc2c681c1393b443534e78a3c3b1456c61370bfbff4163"}
Jul 24 12:20:49.531051 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:49.531012    2566 scope.go:117] "RemoveContainer" containerID="4a01041906baa9df24dc2c681c1393b443534e78a3c3b1456c61370bfbff4163"
Jul 24 12:20:50.088215 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.088176    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:50.182958 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.182924    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:50.536221 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.536188    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n" event={"ID":"f810f78d-7227-4bd0-aab4-194ced389280","Type":"ContainerStarted","Data":"75973f408411cf48d7ee9fa386fe4f14a995bab9a8f24376a5be21f0bdcfc279"}
Jul 24 12:20:50.555259 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.555223    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-847cc74b65-87hbw" podUID="a1b2b5c6-4978-40e1-ad9a-a39876b680a7" containerName="registry" containerID="cri-o://8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e" gracePeriod=30
Jul 24 12:20:50.853079 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.853049    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:20:50.982683 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.982657    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-ca-trust-extracted\") pod \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") "
Jul 24 12:20:50.982683 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.982685    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-certificates\") pod \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") "
Jul 24 12:20:50.982895 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.982706    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-trusted-ca\") pod \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") "
Jul 24 12:20:50.982895 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.982724    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls\") pod \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") "
Jul 24 12:20:50.982895 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.982792    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-image-registry-private-configuration\") pod \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") "
Jul 24 12:20:50.982895 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.982814    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-installation-pull-secrets\") pod \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") "
Jul 24 12:20:50.982895 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.982830    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-bound-sa-token\") pod \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") "
Jul 24 12:20:50.982895 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.982851    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsl7m\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-kube-api-access-wsl7m\") pod \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\" (UID: \"a1b2b5c6-4978-40e1-ad9a-a39876b680a7\") "
Jul 24 12:20:50.983215 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.983167    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a1b2b5c6-4978-40e1-ad9a-a39876b680a7" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:20:50.983270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.983221    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a1b2b5c6-4978-40e1-ad9a-a39876b680a7" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:20:50.985587 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.985538    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a1b2b5c6-4978-40e1-ad9a-a39876b680a7" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:20:50.985715 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.985587    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a1b2b5c6-4978-40e1-ad9a-a39876b680a7" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:20:50.985715 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.985659    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a1b2b5c6-4978-40e1-ad9a-a39876b680a7" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:20:50.985811 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.985728    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-kube-api-access-wsl7m" (OuterVolumeSpecName: "kube-api-access-wsl7m") pod "a1b2b5c6-4978-40e1-ad9a-a39876b680a7" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7"). InnerVolumeSpecName "kube-api-access-wsl7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:20:50.985811 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.985748    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a1b2b5c6-4978-40e1-ad9a-a39876b680a7" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:20:50.991591 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:50.991568    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a1b2b5c6-4978-40e1-ad9a-a39876b680a7" (UID: "a1b2b5c6-4978-40e1-ad9a-a39876b680a7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:20:51.084336 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.084257    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-image-registry-private-configuration\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:20:51.084336 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.084285    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-installation-pull-secrets\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:20:51.084336 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.084300    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-bound-sa-token\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:20:51.084336 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.084314    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsl7m\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-kube-api-access-wsl7m\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:20:51.084336 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.084325    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-ca-trust-extracted\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:20:51.084336 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.084339    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-certificates\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:20:51.084606 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.084351    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-trusted-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:20:51.084606 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.084363    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1b2b5c6-4978-40e1-ad9a-a39876b680a7-registry-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:20:51.540812 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.540774    2566 generic.go:356] "Generic (PLEG): container finished" podID="a1b2b5c6-4978-40e1-ad9a-a39876b680a7" containerID="8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e" exitCode=0
Jul 24 12:20:51.541271 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.540826    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847cc74b65-87hbw"
Jul 24 12:20:51.541271 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.540865    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-847cc74b65-87hbw" event={"ID":"a1b2b5c6-4978-40e1-ad9a-a39876b680a7","Type":"ContainerDied","Data":"8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e"}
Jul 24 12:20:51.541271 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.540923    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-847cc74b65-87hbw" event={"ID":"a1b2b5c6-4978-40e1-ad9a-a39876b680a7","Type":"ContainerDied","Data":"03a8e74bfd7e5486b84c9cf240a657a5f0348707db1178f546e774c98e8cc21f"}
Jul 24 12:20:51.541271 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.540944    2566 scope.go:117] "RemoveContainer" containerID="8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e"
Jul 24 12:20:51.549510 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.549492    2566 scope.go:117] "RemoveContainer" containerID="8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e"
Jul 24 12:20:51.549753 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:20:51.549730    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e\": container with ID starting with 8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e not found: ID does not exist" containerID="8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e"
Jul 24 12:20:51.549846 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.549756    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e"} err="failed to get container status \"8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e\": rpc error: code = NotFound desc = could not find container \"8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e\": container with ID starting with 8a1b2e171e1fc20d3095c6c7b1b496200d2d6cdcbbacc73c9612755a6924969e not found: ID does not exist"
Jul 24 12:20:51.561004 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.560983    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-847cc74b65-87hbw"]
Jul 24 12:20:51.565030 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:51.564997    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-847cc74b65-87hbw"]
Jul 24 12:20:52.973643 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:52.973612    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1b2b5c6-4978-40e1-ad9a-a39876b680a7" path="/var/lib/kubelet/pods/a1b2b5c6-4978-40e1-ad9a-a39876b680a7/volumes"
Jul 24 12:20:56.731773 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:56.731742    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:20:56.734224 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:56.734197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7585202d-e3a1-46b8-90e8-316041c7e09f-metrics-certs\") pod \"network-metrics-daemon-zdv26\" (UID: \"7585202d-e3a1-46b8-90e8-316041c7e09f\") " pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:20:56.910264 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:56.910237    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zhxrp\""
Jul 24 12:20:56.917978 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:56.917964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zdv26"
Jul 24 12:20:57.070451 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:57.070291    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zdv26"]
Jul 24 12:20:57.072451 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:20:57.072424    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7585202d_e3a1_46b8_90e8_316041c7e09f.slice/crio-59eadf4465c342a16d714ddb5dc10d46fc8223068ca075b6907ca6342c718ae4 WatchSource:0}: Error finding container 59eadf4465c342a16d714ddb5dc10d46fc8223068ca075b6907ca6342c718ae4: Status 404 returned error can't find the container with id 59eadf4465c342a16d714ddb5dc10d46fc8223068ca075b6907ca6342c718ae4
Jul 24 12:20:57.565409 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:57.565374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zdv26" event={"ID":"7585202d-e3a1-46b8-90e8-316041c7e09f","Type":"ContainerStarted","Data":"59eadf4465c342a16d714ddb5dc10d46fc8223068ca075b6907ca6342c718ae4"}
Jul 24 12:20:58.570864 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:58.570825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zdv26" event={"ID":"7585202d-e3a1-46b8-90e8-316041c7e09f","Type":"ContainerStarted","Data":"50382eab7806ea1a6be40f4411bb2bbcbada249cc687c0323e680f00966381e6"}
Jul 24 12:20:58.571280 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:58.570905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zdv26" event={"ID":"7585202d-e3a1-46b8-90e8-316041c7e09f","Type":"ContainerStarted","Data":"0d8c6850c7c8210ef90ff2eac6b63428e11d5ff8fc9181f6ebee3dac54263399"}
Jul 24 12:20:58.590790 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:58.590744    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zdv26" podStartSLOduration=130.71341894 podStartE2EDuration="2m11.590729187s" totalImagesPullingTime="877.310247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:58.588407399 +0000 UTC m=+132.116154458" watchObservedRunningTime="2026-07-24 12:20:58.590729187 +0000 UTC m=+132.118476240"
Jul 24 12:20:58.760703 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:58.760670    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:58.816031 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:58.815988    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:20:59.575865 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:59.575831    2566 generic.go:356] "Generic (PLEG): container finished" podID="45ae8a84-d47e-461b-bd9a-6d076a5b94f0" containerID="8527b115d5e4127ef156b209c6b53bb2ed66fc25b1aa279e727bbcbdfad1bc6c" exitCode=0
Jul 24 12:20:59.576251 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:59.575908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zlm49" event={"ID":"45ae8a84-d47e-461b-bd9a-6d076a5b94f0","Type":"ContainerDied","Data":"8527b115d5e4127ef156b209c6b53bb2ed66fc25b1aa279e727bbcbdfad1bc6c"}
Jul 24 12:20:59.576373 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:20:59.576357    2566 scope.go:117] "RemoveContainer" containerID="8527b115d5e4127ef156b209c6b53bb2ed66fc25b1aa279e727bbcbdfad1bc6c"
Jul 24 12:21:00.582340 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:00.582311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zlm49" event={"ID":"45ae8a84-d47e-461b-bd9a-6d076a5b94f0","Type":"ContainerStarted","Data":"9ad5461b977dcee66bce32d8b51148dabb15a2e2b4533e1fc8630054a7c7db44"}
Jul 24 12:21:02.169975 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:02.169936    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx" podUID="362c3d1a-f1d4-45f8-9fbc-b91de7fc284d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 24 12:21:04.599441 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:04.599406    2566 generic.go:356] "Generic (PLEG): container finished" podID="0c42a44d-e70d-4baa-8fd9-5489b77c1e18" containerID="f642750186334f4b4bc82f3f1baab5bbce8d1cee45c30528a996d8684e0dfce6" exitCode=0
Jul 24 12:21:04.599827 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:04.599449    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq" event={"ID":"0c42a44d-e70d-4baa-8fd9-5489b77c1e18","Type":"ContainerDied","Data":"f642750186334f4b4bc82f3f1baab5bbce8d1cee45c30528a996d8684e0dfce6"}
Jul 24 12:21:04.599827 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:04.599723    2566 scope.go:117] "RemoveContainer" containerID="f642750186334f4b4bc82f3f1baab5bbce8d1cee45c30528a996d8684e0dfce6"
Jul 24 12:21:05.097139 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:05.097100    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:05.164461 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:05.164431    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:05.604833 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:05.604799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-sdlcq" event={"ID":"0c42a44d-e70d-4baa-8fd9-5489b77c1e18","Type":"ContainerStarted","Data":"f17fa16bb34616b43013fbac68a4b24a26ddc78e69af88e81e47844ac15f2d27"}
Jul 24 12:21:12.170076 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:12.170033    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx" podUID="362c3d1a-f1d4-45f8-9fbc-b91de7fc284d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 24 12:21:22.169819 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:22.169765    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx" podUID="362c3d1a-f1d4-45f8-9fbc-b91de7fc284d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 24 12:21:22.170317 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:22.169915    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx"
Jul 24 12:21:22.170520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:22.170502    2566 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c49ee9c73fb4a572e1754c6810c398049144fdbd5be80dee4c4e1edcdb27893d"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 24 12:21:22.170557 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:22.170542    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx" podUID="362c3d1a-f1d4-45f8-9fbc-b91de7fc284d" containerName="service-proxy" containerID="cri-o://c49ee9c73fb4a572e1754c6810c398049144fdbd5be80dee4c4e1edcdb27893d" gracePeriod=30
Jul 24 12:21:22.678756 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:22.678721    2566 generic.go:356] "Generic (PLEG): container finished" podID="362c3d1a-f1d4-45f8-9fbc-b91de7fc284d" containerID="c49ee9c73fb4a572e1754c6810c398049144fdbd5be80dee4c4e1edcdb27893d" exitCode=2
Jul 24 12:21:22.678950 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:22.678805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx" event={"ID":"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d","Type":"ContainerDied","Data":"c49ee9c73fb4a572e1754c6810c398049144fdbd5be80dee4c4e1edcdb27893d"}
Jul 24 12:21:22.678950 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:22.678854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56bbb7fb48-x2kvx" event={"ID":"362c3d1a-f1d4-45f8-9fbc-b91de7fc284d","Type":"ContainerStarted","Data":"203776425bf5a1f1585ca95c731550de140d6efa37565d7f51c3702717276106"}
Jul 24 12:21:24.704003 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:24.703967    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:24.766342 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:24.766307    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:31.748550 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:31.748510    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:31.804285 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:31.804244    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:36.630042 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:36.629979    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:36.683610 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:36.683574    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:41.494375 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:41.494279    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:43.114997 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:43.114965    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:43.231947 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:43.231911    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:43.320999 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:43.320964    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:50.121480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:50.121439    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:21:50.217884 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:21:50.217847    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:22:30.360887 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:22:30.360850    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:22:34.811805 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:22:34.811756    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:22:34.904046 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:22:34.903990    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:22:56.608089 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:22:56.608044    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:22:57.444729 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:22:57.444689    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:00.219649 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:00.219610    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:00.284104 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:00.284057    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:01.664735 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:01.664704    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:01.785890 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:01.785850    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:02.472191 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:02.472149    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:02.629548 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:02.629515    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:04.688031 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:04.687968    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:05.063152 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:05.063103    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:06.807347 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:06.807258    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:06.907108 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:06.907074    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:09.702495 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:09.702416    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:09.780827 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:09.780782    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:12.685679 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:12.685646    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:13.229780 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:13.229742    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:13.331039 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:13.330980    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:16.835715 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:16.835668    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:21.098617 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:21.098576    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:26.620451 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:26.620411    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:26.731836 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:26.731791    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:28.582357 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:28.582320    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:36.879295 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:36.879258    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:37.029013 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:37.028938    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:44.698962 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:44.698924    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:44.799928 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:44.799892    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:45.520427 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:45.520389    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:46.285091 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.285053    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"]
Jul 24 12:23:46.285497 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.285353    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b2b5c6-4978-40e1-ad9a-a39876b680a7" containerName="registry"
Jul 24 12:23:46.285497 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.285363    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b2b5c6-4978-40e1-ad9a-a39876b680a7" containerName="registry"
Jul 24 12:23:46.285497 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.285431    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b2b5c6-4978-40e1-ad9a-a39876b680a7" containerName="registry"
Jul 24 12:23:46.288293 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.288273    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.293187 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.293163    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 12:23:46.293187 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.293179    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 12:23:46.293391 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.293241    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 12:23:46.293475 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.293458    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-w2xqh\""
Jul 24 12:23:46.294802 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.294783    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 12:23:46.333442 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.333404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5571d770-b75d-4adf-aa8e-44ad5860e23f-webhook-cert\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.333633 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.333452    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clmfd\" (UniqueName: \"kubernetes.io/projected/5571d770-b75d-4adf-aa8e-44ad5860e23f-kube-api-access-clmfd\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.333633 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.333482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5571d770-b75d-4adf-aa8e-44ad5860e23f-manifests\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.333633 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.333584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5571d770-b75d-4adf-aa8e-44ad5860e23f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.333747 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.333646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5571d770-b75d-4adf-aa8e-44ad5860e23f-tmp\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.402637 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.402598    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"]
Jul 24 12:23:46.426150 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.426110    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:46.434209 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.434172    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5571d770-b75d-4adf-aa8e-44ad5860e23f-webhook-cert\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.434394 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.434226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clmfd\" (UniqueName: \"kubernetes.io/projected/5571d770-b75d-4adf-aa8e-44ad5860e23f-kube-api-access-clmfd\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.434394 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.434268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5571d770-b75d-4adf-aa8e-44ad5860e23f-manifests\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.434394 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.434315    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5571d770-b75d-4adf-aa8e-44ad5860e23f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.434394 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.434361    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5571d770-b75d-4adf-aa8e-44ad5860e23f-tmp\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.434776 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.434746    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5571d770-b75d-4adf-aa8e-44ad5860e23f-manifests\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.436826 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.436802    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5571d770-b75d-4adf-aa8e-44ad5860e23f-tmp\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.437237 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.437215    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5571d770-b75d-4adf-aa8e-44ad5860e23f-webhook-cert\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.437303 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.437215    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5571d770-b75d-4adf-aa8e-44ad5860e23f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.463334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.463302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clmfd\" (UniqueName: \"kubernetes.io/projected/5571d770-b75d-4adf-aa8e-44ad5860e23f-kube-api-access-clmfd\") pod \"opendatahub-operator-controller-manager-565c4d787c-skcjs\" (UID: \"5571d770-b75d-4adf-aa8e-44ad5860e23f\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.560855 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.560758    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:46.599225 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.599186    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:46.732778 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.732720    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:46.763453 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.763425    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"]
Jul 24 12:23:46.765562 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:23:46.765533    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5571d770_b75d_4adf_aa8e_44ad5860e23f.slice/crio-7055fba12d8dde28deea59aadd14c7b829194d53f0b352dfdd7594c3df787c5d WatchSource:0}: Error finding container 7055fba12d8dde28deea59aadd14c7b829194d53f0b352dfdd7594c3df787c5d: Status 404 returned error can't find the container with id 7055fba12d8dde28deea59aadd14c7b829194d53f0b352dfdd7594c3df787c5d
Jul 24 12:23:46.896172 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.896084    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:23:46.901344 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.901307    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:46.903310 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.903288    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:23:46.908623 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:46.908596    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 12:23:47.036746 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:47.036709    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:47.227266 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:47.227167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs" event={"ID":"5571d770-b75d-4adf-aa8e-44ad5860e23f","Type":"ContainerStarted","Data":"7055fba12d8dde28deea59aadd14c7b829194d53f0b352dfdd7594c3df787c5d"}
Jul 24 12:23:47.289253 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:47.289208    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:47.433207 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:47.433148    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:47.570671 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:47.570629    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:50.246811 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:50.246771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs" event={"ID":"5571d770-b75d-4adf-aa8e-44ad5860e23f","Type":"ContainerStarted","Data":"23f8c87694c06a7c28187574627378b32141589594b47bf67998e6e63962cd7c"}
Jul 24 12:23:51.252252 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:51.252206    2566 generic.go:356] "Generic (PLEG): container finished" podID="5571d770-b75d-4adf-aa8e-44ad5860e23f" containerID="23f8c87694c06a7c28187574627378b32141589594b47bf67998e6e63962cd7c" exitCode=0
Jul 24 12:23:51.252827 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:51.252287    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs" event={"ID":"5571d770-b75d-4adf-aa8e-44ad5860e23f","Type":"ContainerDied","Data":"23f8c87694c06a7c28187574627378b32141589594b47bf67998e6e63962cd7c"}
Jul 24 12:23:51.252886 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:51.252844    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 12:23:52.260242 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:52.260204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs" event={"ID":"5571d770-b75d-4adf-aa8e-44ad5860e23f","Type":"ContainerStarted","Data":"7dc4ed8a49ec78e626613f358326f77ab05481e87ac92a53f512c0abc41ddf86"}
Jul 24 12:23:52.260674 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:52.260327    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:23:52.290919 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:52.290847    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs" podStartSLOduration=1.773832333 podStartE2EDuration="6.290825284s" totalImagesPullingTime="3.425117086s" totalInitContainerRuntime="1.091875865s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:23:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:23:52.290639576 +0000 UTC m=+305.818386631" watchObservedRunningTime="2026-07-24 12:23:52.290825284 +0000 UTC m=+305.818572338"
Jul 24 12:23:57.573771 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:57.573729    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:57.712141 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:57.712106    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:58.904480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:58.904437    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:23:59.050588 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:23:59.050549    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:01.181417 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:01.181373    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:01.282580 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:01.282535    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:03.266917 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:03.266885    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-skcjs"
Jul 24 12:24:06.335828 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:06.335784    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:06.797897 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:06.797850    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:07.729084 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:07.729041    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-crq7d"]
Jul 24 12:24:07.732588 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:07.732564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-crq7d"
Jul 24 12:24:07.745174 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:07.745133    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 24 12:24:07.745353 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:07.745174    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4m98z\""
Jul 24 12:24:07.791055 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:07.790992    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-crq7d"]
Jul 24 12:24:07.822093 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:07.822049    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k5d8\" (UniqueName: \"kubernetes.io/projected/e976a183-bb5d-4441-baa3-2fe8718d5ac1-kube-api-access-5k5d8\") pod \"odh-model-controller-7998c67dc4-crq7d\" (UID: \"e976a183-bb5d-4441-baa3-2fe8718d5ac1\") " pod="opendatahub/odh-model-controller-7998c67dc4-crq7d"
Jul 24 12:24:07.822275 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:07.822111    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e976a183-bb5d-4441-baa3-2fe8718d5ac1-cert\") pod \"odh-model-controller-7998c67dc4-crq7d\" (UID: \"e976a183-bb5d-4441-baa3-2fe8718d5ac1\") " pod="opendatahub/odh-model-controller-7998c67dc4-crq7d"
Jul 24 12:24:07.923135 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:07.923092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k5d8\" (UniqueName: \"kubernetes.io/projected/e976a183-bb5d-4441-baa3-2fe8718d5ac1-kube-api-access-5k5d8\") pod \"odh-model-controller-7998c67dc4-crq7d\" (UID: \"e976a183-bb5d-4441-baa3-2fe8718d5ac1\") " pod="opendatahub/odh-model-controller-7998c67dc4-crq7d"
Jul 24 12:24:07.923328 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:07.923169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e976a183-bb5d-4441-baa3-2fe8718d5ac1-cert\") pod \"odh-model-controller-7998c67dc4-crq7d\" (UID: \"e976a183-bb5d-4441-baa3-2fe8718d5ac1\") " pod="opendatahub/odh-model-controller-7998c67dc4-crq7d"
Jul 24 12:24:07.923328 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:24:07.923269    2566 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 24 12:24:07.923403 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:24:07.923341    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e976a183-bb5d-4441-baa3-2fe8718d5ac1-cert podName:e976a183-bb5d-4441-baa3-2fe8718d5ac1 nodeName:}" failed. No retries permitted until 2026-07-24 12:24:08.423325599 +0000 UTC m=+321.951072634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e976a183-bb5d-4441-baa3-2fe8718d5ac1-cert") pod "odh-model-controller-7998c67dc4-crq7d" (UID: "e976a183-bb5d-4441-baa3-2fe8718d5ac1") : secret "odh-model-controller-webhook-cert" not found
Jul 24 12:24:07.946554 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:07.946517    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k5d8\" (UniqueName: \"kubernetes.io/projected/e976a183-bb5d-4441-baa3-2fe8718d5ac1-kube-api-access-5k5d8\") pod \"odh-model-controller-7998c67dc4-crq7d\" (UID: \"e976a183-bb5d-4441-baa3-2fe8718d5ac1\") " pod="opendatahub/odh-model-controller-7998c67dc4-crq7d"
Jul 24 12:24:08.428570 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.428509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e976a183-bb5d-4441-baa3-2fe8718d5ac1-cert\") pod \"odh-model-controller-7998c67dc4-crq7d\" (UID: \"e976a183-bb5d-4441-baa3-2fe8718d5ac1\") " pod="opendatahub/odh-model-controller-7998c67dc4-crq7d"
Jul 24 12:24:08.431411 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.431374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e976a183-bb5d-4441-baa3-2fe8718d5ac1-cert\") pod \"odh-model-controller-7998c67dc4-crq7d\" (UID: \"e976a183-bb5d-4441-baa3-2fe8718d5ac1\") " pod="opendatahub/odh-model-controller-7998c67dc4-crq7d"
Jul 24 12:24:08.461771 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.461735    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:08.643300 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.643257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-crq7d"
Jul 24 12:24:08.809387 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.809336    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-crq7d"]
Jul 24 12:24:08.813148 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:24:08.813109    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode976a183_bb5d_4441_baa3_2fe8718d5ac1.slice/crio-d73e72aa6094e18130ec0f18462476e6daa9c3336b154b469d9465ce7e135884 WatchSource:0}: Error finding container d73e72aa6094e18130ec0f18462476e6daa9c3336b154b469d9465ce7e135884: Status 404 returned error can't find the container with id d73e72aa6094e18130ec0f18462476e6daa9c3336b154b469d9465ce7e135884
Jul 24 12:24:08.823251 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.823210    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:08.867074 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.867009    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qwnnn"]
Jul 24 12:24:08.871599 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.871579    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn"
Jul 24 12:24:08.877405 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.877370    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 24 12:24:08.877405 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.877399    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-bz5rm\""
Jul 24 12:24:08.897668 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.897633    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qwnnn"]
Jul 24 12:24:08.934299 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.934256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpgbq\" (UniqueName: \"kubernetes.io/projected/fa378298-04b3-4401-8fff-0256174b7a4c-kube-api-access-qpgbq\") pod \"model-serving-api-54d89bd5fc-qwnnn\" (UID: \"fa378298-04b3-4401-8fff-0256174b7a4c\") " pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn"
Jul 24 12:24:08.934494 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.934357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa378298-04b3-4401-8fff-0256174b7a4c-tls-certs\") pod \"model-serving-api-54d89bd5fc-qwnnn\" (UID: \"fa378298-04b3-4401-8fff-0256174b7a4c\") " pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn"
Jul 24 12:24:08.983349 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:08.983255    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:09.035703 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:09.035652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgbq\" (UniqueName: \"kubernetes.io/projected/fa378298-04b3-4401-8fff-0256174b7a4c-kube-api-access-qpgbq\") pod \"model-serving-api-54d89bd5fc-qwnnn\" (UID: \"fa378298-04b3-4401-8fff-0256174b7a4c\") " pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn"
Jul 24 12:24:09.035907 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:09.035750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa378298-04b3-4401-8fff-0256174b7a4c-tls-certs\") pod \"model-serving-api-54d89bd5fc-qwnnn\" (UID: \"fa378298-04b3-4401-8fff-0256174b7a4c\") " pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn"
Jul 24 12:24:09.038584 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:09.038558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa378298-04b3-4401-8fff-0256174b7a4c-tls-certs\") pod \"model-serving-api-54d89bd5fc-qwnnn\" (UID: \"fa378298-04b3-4401-8fff-0256174b7a4c\") " pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn"
Jul 24 12:24:09.048366 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:09.048340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgbq\" (UniqueName: \"kubernetes.io/projected/fa378298-04b3-4401-8fff-0256174b7a4c-kube-api-access-qpgbq\") pod \"model-serving-api-54d89bd5fc-qwnnn\" (UID: \"fa378298-04b3-4401-8fff-0256174b7a4c\") " pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn"
Jul 24 12:24:09.182632 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:09.182589    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn"
Jul 24 12:24:09.335523 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:09.335457    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-crq7d" event={"ID":"e976a183-bb5d-4441-baa3-2fe8718d5ac1","Type":"ContainerStarted","Data":"d73e72aa6094e18130ec0f18462476e6daa9c3336b154b469d9465ce7e135884"}
Jul 24 12:24:09.344804 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:09.344778    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qwnnn"]
Jul 24 12:24:09.347285 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:24:09.347243    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa378298_04b3_4401_8fff_0256174b7a4c.slice/crio-9e3d5b9c650a45eb466072bd17da63660d839a490984626829d5504a37be3c0d WatchSource:0}: Error finding container 9e3d5b9c650a45eb466072bd17da63660d839a490984626829d5504a37be3c0d: Status 404 returned error can't find the container with id 9e3d5b9c650a45eb466072bd17da63660d839a490984626829d5504a37be3c0d
Jul 24 12:24:10.343987 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:10.343901    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn" event={"ID":"fa378298-04b3-4401-8fff-0256174b7a4c","Type":"ContainerStarted","Data":"9e3d5b9c650a45eb466072bd17da63660d839a490984626829d5504a37be3c0d"}
Jul 24 12:24:12.354131 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:12.354085    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-crq7d" event={"ID":"e976a183-bb5d-4441-baa3-2fe8718d5ac1","Type":"ContainerStarted","Data":"16e3e33ab4ef96b71bd201c0882ea04ac893d65cfc537c7d64df681b324eb89e"}
Jul 24 12:24:12.354616 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:12.354187    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-crq7d"
Jul 24 12:24:12.355642 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:12.355616    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn" event={"ID":"fa378298-04b3-4401-8fff-0256174b7a4c","Type":"ContainerStarted","Data":"030b4e4ecd5898461589317d8c21f3a8b137b021a78154fb795acde4be831e51"}
Jul 24 12:24:12.355772 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:12.355757    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn"
Jul 24 12:24:12.375600 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:12.375503    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-crq7d" podStartSLOduration=1.987640527 podStartE2EDuration="5.375481414s" totalImagesPullingTime="3.387840887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:24:12.373978798 +0000 UTC m=+325.901725857" watchObservedRunningTime="2026-07-24 12:24:12.375481414 +0000 UTC m=+325.903228472"
Jul 24 12:24:12.409045 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:12.408967    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn" podStartSLOduration=1.55817552 podStartE2EDuration="4.408944259s" totalImagesPullingTime="2.850768739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:24:12.408702965 +0000 UTC m=+325.936450020" watchObservedRunningTime="2026-07-24 12:24:12.408944259 +0000 UTC m=+325.936691315"
Jul 24 12:24:17.939604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:17.939546    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:18.024336 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.024297    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"]
Jul 24 12:24:18.027479 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.027462    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"
Jul 24 12:24:18.033280 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.033259    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 12:24:18.033426 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.033259    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 12:24:18.071189 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.071141    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"]
Jul 24 12:24:18.118634 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.118600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrfkr\" (UniqueName: \"kubernetes.io/projected/43656476-8328-40be-9c33-1e1cd7f640f5-kube-api-access-hrfkr\") pod \"kube-auth-proxy-5bfc95c995-zn967\" (UID: \"43656476-8328-40be-9c33-1e1cd7f640f5\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"
Jul 24 12:24:18.118910 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.118668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43656476-8328-40be-9c33-1e1cd7f640f5-tls-certs\") pod \"kube-auth-proxy-5bfc95c995-zn967\" (UID: \"43656476-8328-40be-9c33-1e1cd7f640f5\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"
Jul 24 12:24:18.118910 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.118722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43656476-8328-40be-9c33-1e1cd7f640f5-tmp\") pod \"kube-auth-proxy-5bfc95c995-zn967\" (UID: \"43656476-8328-40be-9c33-1e1cd7f640f5\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"
Jul 24 12:24:18.219339 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.219234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43656476-8328-40be-9c33-1e1cd7f640f5-tls-certs\") pod \"kube-auth-proxy-5bfc95c995-zn967\" (UID: \"43656476-8328-40be-9c33-1e1cd7f640f5\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"
Jul 24 12:24:18.219339 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.219299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43656476-8328-40be-9c33-1e1cd7f640f5-tmp\") pod \"kube-auth-proxy-5bfc95c995-zn967\" (UID: \"43656476-8328-40be-9c33-1e1cd7f640f5\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"
Jul 24 12:24:18.219583 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.219365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrfkr\" (UniqueName: \"kubernetes.io/projected/43656476-8328-40be-9c33-1e1cd7f640f5-kube-api-access-hrfkr\") pod \"kube-auth-proxy-5bfc95c995-zn967\" (UID: \"43656476-8328-40be-9c33-1e1cd7f640f5\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"
Jul 24 12:24:18.219583 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:24:18.219381    2566 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 24 12:24:18.219583 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:24:18.219460    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43656476-8328-40be-9c33-1e1cd7f640f5-tls-certs podName:43656476-8328-40be-9c33-1e1cd7f640f5 nodeName:}" failed. No retries permitted until 2026-07-24 12:24:18.719439621 +0000 UTC m=+332.247186654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/43656476-8328-40be-9c33-1e1cd7f640f5-tls-certs") pod "kube-auth-proxy-5bfc95c995-zn967" (UID: "43656476-8328-40be-9c33-1e1cd7f640f5") : secret "kube-auth-proxy-tls" not found
Jul 24 12:24:18.221703 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.221679    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43656476-8328-40be-9c33-1e1cd7f640f5-tmp\") pod \"kube-auth-proxy-5bfc95c995-zn967\" (UID: \"43656476-8328-40be-9c33-1e1cd7f640f5\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"
Jul 24 12:24:18.259117 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.259077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrfkr\" (UniqueName: \"kubernetes.io/projected/43656476-8328-40be-9c33-1e1cd7f640f5-kube-api-access-hrfkr\") pod \"kube-auth-proxy-5bfc95c995-zn967\" (UID: \"43656476-8328-40be-9c33-1e1cd7f640f5\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"
Jul 24 12:24:18.508168 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.508131    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:18.724790 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.724733    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43656476-8328-40be-9c33-1e1cd7f640f5-tls-certs\") pod \"kube-auth-proxy-5bfc95c995-zn967\" (UID: \"43656476-8328-40be-9c33-1e1cd7f640f5\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"
Jul 24 12:24:18.727401 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.727374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43656476-8328-40be-9c33-1e1cd7f640f5-tls-certs\") pod \"kube-auth-proxy-5bfc95c995-zn967\" (UID: \"43656476-8328-40be-9c33-1e1cd7f640f5\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"
Jul 24 12:24:18.937499 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:18.937393    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"
Jul 24 12:24:19.145913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:19.145751    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bfc95c995-zn967"]
Jul 24 12:24:19.148727 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:24:19.148701    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43656476_8328_40be_9c33_1e1cd7f640f5.slice/crio-edebf75a5c5fe0b7b4723b1f6d436912e2bb16bf09c462f95c450a2baf987dca WatchSource:0}: Error finding container edebf75a5c5fe0b7b4723b1f6d436912e2bb16bf09c462f95c450a2baf987dca: Status 404 returned error can't find the container with id edebf75a5c5fe0b7b4723b1f6d436912e2bb16bf09c462f95c450a2baf987dca
Jul 24 12:24:19.388560 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:19.388523    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967" event={"ID":"43656476-8328-40be-9c33-1e1cd7f640f5","Type":"ContainerStarted","Data":"edebf75a5c5fe0b7b4723b1f6d436912e2bb16bf09c462f95c450a2baf987dca"}
Jul 24 12:24:21.252706 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.252660    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:21.411329 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.411289    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"]
Jul 24 12:24:21.414999 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.414964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"
Jul 24 12:24:21.422337 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.422313    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 24 12:24:21.423763 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.423729    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-j8fjd\""
Jul 24 12:24:21.438050 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.437952    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"]
Jul 24 12:24:21.440035 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.439977    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"]
Jul 24 12:24:21.443457 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.443431    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"
Jul 24 12:24:21.448182 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.448156    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 24 12:24:21.448360 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.448291    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vbjlq\""
Jul 24 12:24:21.451179 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.451139    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:21.461411 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.461300    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"]
Jul 24 12:24:21.556333 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.556238    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec482164-b0bd-4a38-993b-489f91fb33d0-cert\") pod \"kserve-controller-manager-d6f4b65b8-8nn7z\" (UID: \"ec482164-b0bd-4a38-993b-489f91fb33d0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"
Jul 24 12:24:21.556333 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.556300    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5pzq\" (UniqueName: \"kubernetes.io/projected/ec482164-b0bd-4a38-993b-489f91fb33d0-kube-api-access-z5pzq\") pod \"kserve-controller-manager-d6f4b65b8-8nn7z\" (UID: \"ec482164-b0bd-4a38-993b-489f91fb33d0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"
Jul 24 12:24:21.556556 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.556377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-649rr\" (UniqueName: \"kubernetes.io/projected/eabaebe5-9634-4a47-a6d8-7fc2d5c980b8-kube-api-access-649rr\") pod \"llmisvc-controller-manager-777c6bd7d5-jl9d4\" (UID: \"eabaebe5-9634-4a47-a6d8-7fc2d5c980b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"
Jul 24 12:24:21.556556 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.556456    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eabaebe5-9634-4a47-a6d8-7fc2d5c980b8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jl9d4\" (UID: \"eabaebe5-9634-4a47-a6d8-7fc2d5c980b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"
Jul 24 12:24:21.562981 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.562942    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:21.657837 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.657800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eabaebe5-9634-4a47-a6d8-7fc2d5c980b8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jl9d4\" (UID: \"eabaebe5-9634-4a47-a6d8-7fc2d5c980b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"
Jul 24 12:24:21.658089 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.657892    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec482164-b0bd-4a38-993b-489f91fb33d0-cert\") pod \"kserve-controller-manager-d6f4b65b8-8nn7z\" (UID: \"ec482164-b0bd-4a38-993b-489f91fb33d0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"
Jul 24 12:24:21.658089 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.657932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5pzq\" (UniqueName: \"kubernetes.io/projected/ec482164-b0bd-4a38-993b-489f91fb33d0-kube-api-access-z5pzq\") pod \"kserve-controller-manager-d6f4b65b8-8nn7z\" (UID: \"ec482164-b0bd-4a38-993b-489f91fb33d0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"
Jul 24 12:24:21.658089 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:24:21.657941    2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 24 12:24:21.658089 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.657982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-649rr\" (UniqueName: \"kubernetes.io/projected/eabaebe5-9634-4a47-a6d8-7fc2d5c980b8-kube-api-access-649rr\") pod \"llmisvc-controller-manager-777c6bd7d5-jl9d4\" (UID: \"eabaebe5-9634-4a47-a6d8-7fc2d5c980b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"
Jul 24 12:24:21.658089 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:24:21.658049    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eabaebe5-9634-4a47-a6d8-7fc2d5c980b8-cert podName:eabaebe5-9634-4a47-a6d8-7fc2d5c980b8 nodeName:}" failed. No retries permitted until 2026-07-24 12:24:22.158010459 +0000 UTC m=+335.685757505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/eabaebe5-9634-4a47-a6d8-7fc2d5c980b8-cert") pod "llmisvc-controller-manager-777c6bd7d5-jl9d4" (UID: "eabaebe5-9634-4a47-a6d8-7fc2d5c980b8") : secret "llmisvc-webhook-server-cert" not found
Jul 24 12:24:21.660961 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.660936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec482164-b0bd-4a38-993b-489f91fb33d0-cert\") pod \"kserve-controller-manager-d6f4b65b8-8nn7z\" (UID: \"ec482164-b0bd-4a38-993b-489f91fb33d0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"
Jul 24 12:24:21.676616 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.676583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5pzq\" (UniqueName: \"kubernetes.io/projected/ec482164-b0bd-4a38-993b-489f91fb33d0-kube-api-access-z5pzq\") pod \"kserve-controller-manager-d6f4b65b8-8nn7z\" (UID: \"ec482164-b0bd-4a38-993b-489f91fb33d0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"
Jul 24 12:24:21.680064 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.680032    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-649rr\" (UniqueName: \"kubernetes.io/projected/eabaebe5-9634-4a47-a6d8-7fc2d5c980b8-kube-api-access-649rr\") pod \"llmisvc-controller-manager-777c6bd7d5-jl9d4\" (UID: \"eabaebe5-9634-4a47-a6d8-7fc2d5c980b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"
Jul 24 12:24:21.727798 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.727757    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"
Jul 24 12:24:21.838205 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.838112    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:21.973461 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.973433    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"]
Jul 24 12:24:21.978351 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.978326    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:21.996792 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.996595    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 24 12:24:21.996792 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.996595    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 24 12:24:21.996792 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.996595    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 24 12:24:21.997346 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.997321    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 24 12:24:21.997546 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:21.997527    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n5pts\""
Jul 24 12:24:22.018797 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.018760    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"]
Jul 24 12:24:22.066895 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.066862    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"]
Jul 24 12:24:22.070001 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.069968    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:22.077996 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:24:22.077973    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec482164_b0bd_4a38_993b_489f91fb33d0.slice/crio-c6cc7199f83ef298513bac433cbd44dfe1338968011eab9cb7241073c932b50a WatchSource:0}: Error finding container c6cc7199f83ef298513bac433cbd44dfe1338968011eab9cb7241073c932b50a: Status 404 returned error can't find the container with id c6cc7199f83ef298513bac433cbd44dfe1338968011eab9cb7241073c932b50a
Jul 24 12:24:22.162846 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.162761    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eabaebe5-9634-4a47-a6d8-7fc2d5c980b8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jl9d4\" (UID: \"eabaebe5-9634-4a47-a6d8-7fc2d5c980b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"
Jul 24 12:24:22.162846 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.162799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8674e584-2789-4dfe-b809-9d93b659e554-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.162846 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.162845    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8674e584-2789-4dfe-b809-9d93b659e554-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.163157 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.162865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8674e584-2789-4dfe-b809-9d93b659e554-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.163157 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.162893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8674e584-2789-4dfe-b809-9d93b659e554-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.163157 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.162961    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8674e584-2789-4dfe-b809-9d93b659e554-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.163157 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.163004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwvv9\" (UniqueName: \"kubernetes.io/projected/8674e584-2789-4dfe-b809-9d93b659e554-kube-api-access-dwvv9\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.163157 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.163121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8674e584-2789-4dfe-b809-9d93b659e554-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.165446 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.165427    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eabaebe5-9634-4a47-a6d8-7fc2d5c980b8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jl9d4\" (UID: \"eabaebe5-9634-4a47-a6d8-7fc2d5c980b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"
Jul 24 12:24:22.264492 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.264445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8674e584-2789-4dfe-b809-9d93b659e554-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.264492 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.264512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwvv9\" (UniqueName: \"kubernetes.io/projected/8674e584-2789-4dfe-b809-9d93b659e554-kube-api-access-dwvv9\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.265097 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.264625    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8674e584-2789-4dfe-b809-9d93b659e554-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.265097 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.264667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8674e584-2789-4dfe-b809-9d93b659e554-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.265097 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.264716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8674e584-2789-4dfe-b809-9d93b659e554-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.265097 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.264736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8674e584-2789-4dfe-b809-9d93b659e554-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.265097 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.264762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8674e584-2789-4dfe-b809-9d93b659e554-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.265455 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.265395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8674e584-2789-4dfe-b809-9d93b659e554-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.267081 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.267057    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8674e584-2789-4dfe-b809-9d93b659e554-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.267265 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.267247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8674e584-2789-4dfe-b809-9d93b659e554-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.267311 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.267247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8674e584-2789-4dfe-b809-9d93b659e554-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.267372 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.267356    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8674e584-2789-4dfe-b809-9d93b659e554-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.290818 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.290778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwvv9\" (UniqueName: \"kubernetes.io/projected/8674e584-2789-4dfe-b809-9d93b659e554-kube-api-access-dwvv9\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.294655 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.294624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8674e584-2789-4dfe-b809-9d93b659e554-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wrjx5\" (UID: \"8674e584-2789-4dfe-b809-9d93b659e554\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.323311 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.323268    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:22.356444 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.356408    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"
Jul 24 12:24:22.406417 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.406371    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z" event={"ID":"ec482164-b0bd-4a38-993b-489f91fb33d0","Type":"ContainerStarted","Data":"c6cc7199f83ef298513bac433cbd44dfe1338968011eab9cb7241073c932b50a"}
Jul 24 12:24:22.408713 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.408551    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967" event={"ID":"43656476-8328-40be-9c33-1e1cd7f640f5","Type":"ContainerStarted","Data":"11db0f0675189fa20b10fdc50012c5c1a294790eac70eba5ce76981b3fd52fec"}
Jul 24 12:24:22.469536 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.469482    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bfc95c995-zn967" podStartSLOduration=2.68872205 podStartE2EDuration="5.469462066s" totalImagesPullingTime="2.780740016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:24:22.465984115 +0000 UTC m=+335.993731172" watchObservedRunningTime="2026-07-24 12:24:22.469462066 +0000 UTC m=+335.997209171"
Jul 24 12:24:22.520516 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.520483    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"]
Jul 24 12:24:22.524371 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:24:22.524327    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeabaebe5_9634_4a47_a6d8_7fc2d5c980b8.slice/crio-7670b4f040cc447dc4b075584d9fa084ca0457694d7c85248d38e298b88dadcf WatchSource:0}: Error finding container 7670b4f040cc447dc4b075584d9fa084ca0457694d7c85248d38e298b88dadcf: Status 404 returned error can't find the container with id 7670b4f040cc447dc4b075584d9fa084ca0457694d7c85248d38e298b88dadcf
Jul 24 12:24:22.591813 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.591777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:22.731498 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.731406    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:22.767411 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.767369    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"]
Jul 24 12:24:22.770107 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:24:22.770072    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8674e584_2789_4dfe_b809_9d93b659e554.slice/crio-b2bc80ed7285832ca771e78005140a1d78a346978461fbe970dea5c33fce9ba5 WatchSource:0}: Error finding container b2bc80ed7285832ca771e78005140a1d78a346978461fbe970dea5c33fce9ba5: Status 404 returned error can't find the container with id b2bc80ed7285832ca771e78005140a1d78a346978461fbe970dea5c33fce9ba5
Jul 24 12:24:22.859702 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:22.859667    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:23.021615 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:23.021580    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:23.362844 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:23.362778    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-crq7d"
Jul 24 12:24:23.364763 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:23.364735    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-qwnnn"
Jul 24 12:24:23.418577 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:23.418321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4" event={"ID":"eabaebe5-9634-4a47-a6d8-7fc2d5c980b8","Type":"ContainerStarted","Data":"7670b4f040cc447dc4b075584d9fa084ca0457694d7c85248d38e298b88dadcf"}
Jul 24 12:24:23.420779 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:23.420695    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5" event={"ID":"8674e584-2789-4dfe-b809-9d93b659e554","Type":"ContainerStarted","Data":"b2bc80ed7285832ca771e78005140a1d78a346978461fbe970dea5c33fce9ba5"}
Jul 24 12:24:25.112070 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:25.112010    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:26.534927 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:26.534885    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 12:24:26.535376 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:26.534967    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 12:24:27.446178 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.446136    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5" event={"ID":"8674e584-2789-4dfe-b809-9d93b659e554","Type":"ContainerStarted","Data":"f2a54f12c2f951657145116502a04a5d0103a795754d006cba8807314641887e"}
Jul 24 12:24:27.446401 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.446334    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:27.447786 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.447759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z" event={"ID":"ec482164-b0bd-4a38-993b-489f91fb33d0","Type":"ContainerStarted","Data":"06ede5b1ec6bdd0d3be71b9e4a26e9912e10bb02aceba4dcadbc928fe9788d2b"}
Jul 24 12:24:27.447949 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.447843    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"
Jul 24 12:24:27.448227 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.448149    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-wrjx5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 12:24:27.448227 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.448190    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5" podUID="8674e584-2789-4dfe-b809-9d93b659e554" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 12:24:27.449530 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.449509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4" event={"ID":"eabaebe5-9634-4a47-a6d8-7fc2d5c980b8","Type":"ContainerStarted","Data":"6f6ae2b0fb6fdc895af863a82aa8bed8761f91f7a579c86fae9f46160dcbab97"}
Jul 24 12:24:27.449635 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.449624    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"
Jul 24 12:24:27.477570 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.477508    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5" podStartSLOduration=2.715358517 podStartE2EDuration="6.477489455s" totalImagesPullingTime="3.762130938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:24:27.476210064 +0000 UTC m=+341.003957118" watchObservedRunningTime="2026-07-24 12:24:27.477489455 +0000 UTC m=+341.005236487"
Jul 24 12:24:27.511438 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.511378    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4" podStartSLOduration=2.499927304 podStartE2EDuration="6.511361298s" totalImagesPullingTime="4.011433994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:24:27.511057776 +0000 UTC m=+341.038804830" watchObservedRunningTime="2026-07-24 12:24:27.511361298 +0000 UTC m=+341.039108352"
Jul 24 12:24:27.531975 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.531926    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z" podStartSLOduration=2.080779052 podStartE2EDuration="6.531909106s" totalImagesPullingTime="4.451130054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:24:27.531108325 +0000 UTC m=+341.058855379" watchObservedRunningTime="2026-07-24 12:24:27.531909106 +0000 UTC m=+341.059656159"
Jul 24 12:24:27.789044 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.788987    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:27.918195 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:27.918145    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:28.455768 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:28.455733    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-wrjx5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 12:24:28.455921 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:28.455801    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5" podUID="8674e584-2789-4dfe-b809-9d93b659e554" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 12:24:28.793614 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:28.793581    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:29.040346 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:29.040307    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:29.172335 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:29.172243    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:24:31.455918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:31.455889    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrjx5"
Jul 24 12:24:58.457317 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:58.457284    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jl9d4"
Jul 24 12:24:58.460365 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:24:58.460347    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8nn7z"
Jul 24 12:25:08.130888 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:08.130849    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:08.318580 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:08.318534    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:08.434797 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:08.434704    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:32.148555 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:32.148510    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:35.404427 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:35.404391    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:36.142982 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:36.142949    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:36.973320 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:36.973278    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:37.070129 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:37.070088    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:39.399555 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:39.399523    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:39.538200 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:39.538163    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:39.788686 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:39.788655    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:41.528078 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:41.528011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:42.549043 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:42.548989    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:43.824562 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:43.824527    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:47.224627 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:47.224583    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:48.238354 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:48.238317    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:48.362937 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:48.362897    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:48.570709 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:48.570676    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:48.716045 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:48.715989    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:52.545096 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:52.545060    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:52.642013 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:52.641977    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:52.951112 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:52.951004    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:53.170949 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:53.170913    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:55.190920 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:55.190884    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:55.522940 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:55.522914    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:55.969372 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:55.969286    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:56.069550 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:56.069515    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:25:56.507569 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:25:56.507530    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:00.059221 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:00.059180    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:00.250073 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:00.250033    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:06.764631 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:06.764587    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:07.017735 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:07.017645    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:10.247050 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:10.246924    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:10.345860 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:10.345813    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:10.445972 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:10.445925    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:10.549437 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:10.549336    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:13.274525 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:13.274492    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:13.414660 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:13.414629    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:18.332626 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:18.332594    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:18.478703 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:18.478673    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:18.577035 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:18.576997    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:42.121385 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:42.121353    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:42.223597 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:42.223558    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:48.689089 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:48.689051    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:48.789452 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:48.789413    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:51.447588 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.447554    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"]
Jul 24 12:26:51.450138 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.450114    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"
Jul 24 12:26:51.451656 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.451631    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:51.453238 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.453208    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nl2w7\""
Jul 24 12:26:51.453238 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.453226    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 12:26:51.454668 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.454649    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 12:26:51.454764 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.454686    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 12:26:51.464744 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.464722    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"]
Jul 24 12:26:51.541519 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.541484    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"]
Jul 24 12:26:51.543994 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.543965    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:51.583832 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.583802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5f54\" (UniqueName: \"kubernetes.io/projected/8d4ab000-e47c-4ae3-9d0a-1989c04774ba-kube-api-access-x5f54\") pod \"limitador-limitador-64c94fd64f-gxlj4\" (UID: \"8d4ab000-e47c-4ae3-9d0a-1989c04774ba\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"
Jul 24 12:26:51.583996 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.583844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d4ab000-e47c-4ae3-9d0a-1989c04774ba-config-file\") pod \"limitador-limitador-64c94fd64f-gxlj4\" (UID: \"8d4ab000-e47c-4ae3-9d0a-1989c04774ba\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"
Jul 24 12:26:51.644897 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.644858    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:51.684932 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.684895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5f54\" (UniqueName: \"kubernetes.io/projected/8d4ab000-e47c-4ae3-9d0a-1989c04774ba-kube-api-access-x5f54\") pod \"limitador-limitador-64c94fd64f-gxlj4\" (UID: \"8d4ab000-e47c-4ae3-9d0a-1989c04774ba\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"
Jul 24 12:26:51.684932 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.684939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d4ab000-e47c-4ae3-9d0a-1989c04774ba-config-file\") pod \"limitador-limitador-64c94fd64f-gxlj4\" (UID: \"8d4ab000-e47c-4ae3-9d0a-1989c04774ba\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"
Jul 24 12:26:51.685564 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.685544    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d4ab000-e47c-4ae3-9d0a-1989c04774ba-config-file\") pod \"limitador-limitador-64c94fd64f-gxlj4\" (UID: \"8d4ab000-e47c-4ae3-9d0a-1989c04774ba\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"
Jul 24 12:26:51.695977 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.695955    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5f54\" (UniqueName: \"kubernetes.io/projected/8d4ab000-e47c-4ae3-9d0a-1989c04774ba-kube-api-access-x5f54\") pod \"limitador-limitador-64c94fd64f-gxlj4\" (UID: \"8d4ab000-e47c-4ae3-9d0a-1989c04774ba\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"
Jul 24 12:26:51.761390 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.761353    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"
Jul 24 12:26:51.907497 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:51.907468    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"]
Jul 24 12:26:51.909373 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:26:51.909344    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4ab000_e47c_4ae3_9d0a_1989c04774ba.slice/crio-d69f5dabef0f87bffbf11105db6e49bff812978424ce8e6e27567084428c704f WatchSource:0}: Error finding container d69f5dabef0f87bffbf11105db6e49bff812978424ce8e6e27567084428c704f: Status 404 returned error can't find the container with id d69f5dabef0f87bffbf11105db6e49bff812978424ce8e6e27567084428c704f
Jul 24 12:26:52.091003 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:52.090917    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4" event={"ID":"8d4ab000-e47c-4ae3-9d0a-1989c04774ba","Type":"ContainerStarted","Data":"d69f5dabef0f87bffbf11105db6e49bff812978424ce8e6e27567084428c704f"}
Jul 24 12:26:52.309076 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:52.309036    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:52.376127 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:52.376013    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:26:55.107350 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:55.107312    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4" event={"ID":"8d4ab000-e47c-4ae3-9d0a-1989c04774ba","Type":"ContainerStarted","Data":"ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a"}
Jul 24 12:26:55.107793 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:55.107393    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"
Jul 24 12:26:55.130098 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:26:55.130041    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4" podStartSLOduration=1.614321187 podStartE2EDuration="4.130005211s" totalImagesPullingTime="2.515684024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:26:55.128980239 +0000 UTC m=+488.656727293" watchObservedRunningTime="2026-07-24 12:26:55.130005211 +0000 UTC m=+488.657752303"
Jul 24 12:27:06.112464 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:06.112432    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"
Jul 24 12:27:06.886579 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:06.886546    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"]
Jul 24 12:27:06.886762 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:06.886728    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4" podUID="8d4ab000-e47c-4ae3-9d0a-1989c04774ba" containerName="limitador" containerID="cri-o://ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a" gracePeriod=30
Jul 24 12:27:07.435848 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:07.435815    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"
Jul 24 12:27:07.526302 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:07.526264    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d4ab000-e47c-4ae3-9d0a-1989c04774ba-config-file\") pod \"8d4ab000-e47c-4ae3-9d0a-1989c04774ba\" (UID: \"8d4ab000-e47c-4ae3-9d0a-1989c04774ba\") "
Jul 24 12:27:07.526500 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:07.526363    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5f54\" (UniqueName: \"kubernetes.io/projected/8d4ab000-e47c-4ae3-9d0a-1989c04774ba-kube-api-access-x5f54\") pod \"8d4ab000-e47c-4ae3-9d0a-1989c04774ba\" (UID: \"8d4ab000-e47c-4ae3-9d0a-1989c04774ba\") "
Jul 24 12:27:07.526632 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:07.526608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4ab000-e47c-4ae3-9d0a-1989c04774ba-config-file" (OuterVolumeSpecName: "config-file") pod "8d4ab000-e47c-4ae3-9d0a-1989c04774ba" (UID: "8d4ab000-e47c-4ae3-9d0a-1989c04774ba"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:27:07.528672 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:07.528650    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4ab000-e47c-4ae3-9d0a-1989c04774ba-kube-api-access-x5f54" (OuterVolumeSpecName: "kube-api-access-x5f54") pod "8d4ab000-e47c-4ae3-9d0a-1989c04774ba" (UID: "8d4ab000-e47c-4ae3-9d0a-1989c04774ba"). InnerVolumeSpecName "kube-api-access-x5f54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:27:07.627630 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:07.627594    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5f54\" (UniqueName: \"kubernetes.io/projected/8d4ab000-e47c-4ae3-9d0a-1989c04774ba-kube-api-access-x5f54\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:27:07.627630 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:07.627623    2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d4ab000-e47c-4ae3-9d0a-1989c04774ba-config-file\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:27:08.166663 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:08.166628    2566 generic.go:356] "Generic (PLEG): container finished" podID="8d4ab000-e47c-4ae3-9d0a-1989c04774ba" containerID="ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a" exitCode=0
Jul 24 12:27:08.166832 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:08.166710    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4" event={"ID":"8d4ab000-e47c-4ae3-9d0a-1989c04774ba","Type":"ContainerDied","Data":"ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a"}
Jul 24 12:27:08.166832 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:08.166725    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"
Jul 24 12:27:08.166832 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:08.166763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gxlj4" event={"ID":"8d4ab000-e47c-4ae3-9d0a-1989c04774ba","Type":"ContainerDied","Data":"d69f5dabef0f87bffbf11105db6e49bff812978424ce8e6e27567084428c704f"}
Jul 24 12:27:08.166832 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:08.166788    2566 scope.go:117] "RemoveContainer" containerID="ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a"
Jul 24 12:27:08.176092 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:08.176072    2566 scope.go:117] "RemoveContainer" containerID="ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a"
Jul 24 12:27:08.176394 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:27:08.176377    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a\": container with ID starting with ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a not found: ID does not exist" containerID="ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a"
Jul 24 12:27:08.176459 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:08.176403    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a"} err="failed to get container status \"ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a\": rpc error: code = NotFound desc = could not find container \"ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a\": container with ID starting with ceb10418a86f51e28f942534d13f931163875f051b6d4fbd09ab16af571bed8a not found: ID does not exist"
Jul 24 12:27:08.189150 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:08.189115    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"]
Jul 24 12:27:08.191011 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:08.190993    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gxlj4"]
Jul 24 12:27:08.974898 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:08.974863    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d4ab000-e47c-4ae3-9d0a-1989c04774ba" path="/var/lib/kubelet/pods/8d4ab000-e47c-4ae3-9d0a-1989c04774ba/volumes"
Jul 24 12:27:12.706602 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:12.706565    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:12.839257 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:12.839226    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:13.164826 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.164795    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"]
Jul 24 12:27:13.165166 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.165152    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d4ab000-e47c-4ae3-9d0a-1989c04774ba" containerName="limitador"
Jul 24 12:27:13.165219 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.165168    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d4ab000-e47c-4ae3-9d0a-1989c04774ba" containerName="limitador"
Jul 24 12:27:13.165257 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.165221    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d4ab000-e47c-4ae3-9d0a-1989c04774ba" containerName="limitador"
Jul 24 12:27:13.168162 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.168147    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"
Jul 24 12:27:13.171710 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.171678    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jc4ww\""
Jul 24 12:27:13.171848 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.171685    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 24 12:27:13.171848 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.171766    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 12:27:13.172995 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.172971    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 12:27:13.178770 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.178750    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"]
Jul 24 12:27:13.217198 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.217165    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:13.274256 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.274226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdt4z\" (UniqueName: \"kubernetes.io/projected/d15e8048-1a41-4264-a75a-11389fda4254-kube-api-access-cdt4z\") pod \"postgres-69fb4594cf-jdvqd\" (UID: \"d15e8048-1a41-4264-a75a-11389fda4254\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"
Jul 24 12:27:13.274256 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.274261    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d15e8048-1a41-4264-a75a-11389fda4254-data\") pod \"postgres-69fb4594cf-jdvqd\" (UID: \"d15e8048-1a41-4264-a75a-11389fda4254\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"
Jul 24 12:27:13.374644 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.374616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdt4z\" (UniqueName: \"kubernetes.io/projected/d15e8048-1a41-4264-a75a-11389fda4254-kube-api-access-cdt4z\") pod \"postgres-69fb4594cf-jdvqd\" (UID: \"d15e8048-1a41-4264-a75a-11389fda4254\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"
Jul 24 12:27:13.374644 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.374648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d15e8048-1a41-4264-a75a-11389fda4254-data\") pod \"postgres-69fb4594cf-jdvqd\" (UID: \"d15e8048-1a41-4264-a75a-11389fda4254\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"
Jul 24 12:27:13.375009 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.374993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d15e8048-1a41-4264-a75a-11389fda4254-data\") pod \"postgres-69fb4594cf-jdvqd\" (UID: \"d15e8048-1a41-4264-a75a-11389fda4254\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"
Jul 24 12:27:13.384319 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.384293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdt4z\" (UniqueName: \"kubernetes.io/projected/d15e8048-1a41-4264-a75a-11389fda4254-kube-api-access-cdt4z\") pod \"postgres-69fb4594cf-jdvqd\" (UID: \"d15e8048-1a41-4264-a75a-11389fda4254\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"
Jul 24 12:27:13.479696 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.479604    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"
Jul 24 12:27:13.682537 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:13.682514    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"]
Jul 24 12:27:13.687902 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:27:13.685508    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15e8048_1a41_4264_a75a_11389fda4254.slice/crio-e49c65aa2e65cb4e426add88448ac7e83ddecede2a097c9cf85d497d7235e989 WatchSource:0}: Error finding container e49c65aa2e65cb4e426add88448ac7e83ddecede2a097c9cf85d497d7235e989: Status 404 returned error can't find the container with id e49c65aa2e65cb4e426add88448ac7e83ddecede2a097c9cf85d497d7235e989
Jul 24 12:27:14.193872 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:14.193838    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd" event={"ID":"d15e8048-1a41-4264-a75a-11389fda4254","Type":"ContainerStarted","Data":"e49c65aa2e65cb4e426add88448ac7e83ddecede2a097c9cf85d497d7235e989"}
Jul 24 12:27:18.102807 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:18.102747    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:19.225396 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:19.225366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd" event={"ID":"d15e8048-1a41-4264-a75a-11389fda4254","Type":"ContainerStarted","Data":"7ff47322f2ebfc71a48c656054abbbac6ee39d249f8470cb8be334c129d2e4b4"}
Jul 24 12:27:19.225816 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:19.225496    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"
Jul 24 12:27:19.242074 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:19.241918    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd" podStartSLOduration=1.269046237 podStartE2EDuration="6.241897814s" totalImagesPullingTime="4.972851577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:27:19.241830504 +0000 UTC m=+512.769577559" watchObservedRunningTime="2026-07-24 12:27:19.241897814 +0000 UTC m=+512.769644870"
Jul 24 12:27:25.259552 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:25.259524    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdvqd"
Jul 24 12:27:25.658425 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:25.658338    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:25.764721 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:25.764685    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:25.942553 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:25.942459    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:26.010440 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:26.010398    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:32.321402 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.321364    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:32.336376 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.336347    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67b674788f-b72pf"]
Jul 24 12:27:32.360588 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.360555    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67b674788f-b72pf"]
Jul 24 12:27:32.360759 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.360667    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:32.364611 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.364590    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lb9qp\""
Jul 24 12:27:32.364842 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.364824    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 12:27:32.365780 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.365764    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 12:27:32.402115 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.402080    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:32.453228 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.453191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e6c816c7-b50d-4828-924a-a0928bde88fd-webhook-cert\") pod \"maas-controller-67b674788f-b72pf\" (UID: \"e6c816c7-b50d-4828-924a-a0928bde88fd\") " pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:32.453228 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.453228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdmk\" (UniqueName: \"kubernetes.io/projected/e6c816c7-b50d-4828-924a-a0928bde88fd-kube-api-access-xrdmk\") pod \"maas-controller-67b674788f-b72pf\" (UID: \"e6c816c7-b50d-4828-924a-a0928bde88fd\") " pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:32.499214 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.499180    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cc9bbb694-k48m6"]
Jul 24 12:27:32.501764 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.501744    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:27:32.512649 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.512624    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc9bbb694-k48m6"]
Jul 24 12:27:32.554550 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.554513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e6c816c7-b50d-4828-924a-a0928bde88fd-webhook-cert\") pod \"maas-controller-67b674788f-b72pf\" (UID: \"e6c816c7-b50d-4828-924a-a0928bde88fd\") " pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:32.554550 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.554549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdmk\" (UniqueName: \"kubernetes.io/projected/e6c816c7-b50d-4828-924a-a0928bde88fd-kube-api-access-xrdmk\") pod \"maas-controller-67b674788f-b72pf\" (UID: \"e6c816c7-b50d-4828-924a-a0928bde88fd\") " pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:32.554761 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:27:32.554656    2566 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 24 12:27:32.554761 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:27:32.554716    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6c816c7-b50d-4828-924a-a0928bde88fd-webhook-cert podName:e6c816c7-b50d-4828-924a-a0928bde88fd nodeName:}" failed. No retries permitted until 2026-07-24 12:27:33.054700867 +0000 UTC m=+526.582447901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/e6c816c7-b50d-4828-924a-a0928bde88fd-webhook-cert") pod "maas-controller-67b674788f-b72pf" (UID: "e6c816c7-b50d-4828-924a-a0928bde88fd") : secret "maas-controller-webhook-cert" not found
Jul 24 12:27:32.564417 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.564386    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdmk\" (UniqueName: \"kubernetes.io/projected/e6c816c7-b50d-4828-924a-a0928bde88fd-kube-api-access-xrdmk\") pod \"maas-controller-67b674788f-b72pf\" (UID: \"e6c816c7-b50d-4828-924a-a0928bde88fd\") " pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:32.655375 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.655265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b4aa5c5-13d2-4e20-883d-1d87baffc373-webhook-cert\") pod \"maas-controller-cc9bbb694-k48m6\" (UID: \"6b4aa5c5-13d2-4e20-883d-1d87baffc373\") " pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:27:32.655375 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.655355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcckv\" (UniqueName: \"kubernetes.io/projected/6b4aa5c5-13d2-4e20-883d-1d87baffc373-kube-api-access-tcckv\") pod \"maas-controller-cc9bbb694-k48m6\" (UID: \"6b4aa5c5-13d2-4e20-883d-1d87baffc373\") " pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:27:32.756289 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.756256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b4aa5c5-13d2-4e20-883d-1d87baffc373-webhook-cert\") pod \"maas-controller-cc9bbb694-k48m6\" (UID: \"6b4aa5c5-13d2-4e20-883d-1d87baffc373\") " pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:27:32.756497 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.756340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcckv\" (UniqueName: \"kubernetes.io/projected/6b4aa5c5-13d2-4e20-883d-1d87baffc373-kube-api-access-tcckv\") pod \"maas-controller-cc9bbb694-k48m6\" (UID: \"6b4aa5c5-13d2-4e20-883d-1d87baffc373\") " pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:27:32.756497 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:27:32.756421    2566 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 24 12:27:32.756591 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:27:32.756497    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4aa5c5-13d2-4e20-883d-1d87baffc373-webhook-cert podName:6b4aa5c5-13d2-4e20-883d-1d87baffc373 nodeName:}" failed. No retries permitted until 2026-07-24 12:27:33.256481676 +0000 UTC m=+526.784228712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/6b4aa5c5-13d2-4e20-883d-1d87baffc373-webhook-cert") pod "maas-controller-cc9bbb694-k48m6" (UID: "6b4aa5c5-13d2-4e20-883d-1d87baffc373") : secret "maas-controller-webhook-cert" not found
Jul 24 12:27:32.767007 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:32.766976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcckv\" (UniqueName: \"kubernetes.io/projected/6b4aa5c5-13d2-4e20-883d-1d87baffc373-kube-api-access-tcckv\") pod \"maas-controller-cc9bbb694-k48m6\" (UID: \"6b4aa5c5-13d2-4e20-883d-1d87baffc373\") " pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:27:33.053412 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:33.053370    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:33.059454 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:33.059433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e6c816c7-b50d-4828-924a-a0928bde88fd-webhook-cert\") pod \"maas-controller-67b674788f-b72pf\" (UID: \"e6c816c7-b50d-4828-924a-a0928bde88fd\") " pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:33.061957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:33.061940    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e6c816c7-b50d-4828-924a-a0928bde88fd-webhook-cert\") pod \"maas-controller-67b674788f-b72pf\" (UID: \"e6c816c7-b50d-4828-924a-a0928bde88fd\") " pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:33.261842 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:33.261789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b4aa5c5-13d2-4e20-883d-1d87baffc373-webhook-cert\") pod \"maas-controller-cc9bbb694-k48m6\" (UID: \"6b4aa5c5-13d2-4e20-883d-1d87baffc373\") " pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:27:33.264394 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:33.264374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b4aa5c5-13d2-4e20-883d-1d87baffc373-webhook-cert\") pod \"maas-controller-cc9bbb694-k48m6\" (UID: \"6b4aa5c5-13d2-4e20-883d-1d87baffc373\") " pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:27:33.271218 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:33.271196    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:33.407681 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:33.407653    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67b674788f-b72pf"]
Jul 24 12:27:33.409057 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:27:33.409031    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c816c7_b50d_4828_924a_a0928bde88fd.slice/crio-cc44687b6846745e37b93be812b9a5aedb10ba35d366c0abc461121eda07cef0 WatchSource:0}: Error finding container cc44687b6846745e37b93be812b9a5aedb10ba35d366c0abc461121eda07cef0: Status 404 returned error can't find the container with id cc44687b6846745e37b93be812b9a5aedb10ba35d366c0abc461121eda07cef0
Jul 24 12:27:33.412056 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:33.412005    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:27:33.549459 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:33.549434    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc9bbb694-k48m6"]
Jul 24 12:27:33.550231 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:27:33.550209    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4aa5c5_13d2_4e20_883d_1d87baffc373.slice/crio-9d8522e251394d16e390b9d46d208c0b3aa715200f9b28ff25fc6855be07bc1f WatchSource:0}: Error finding container 9d8522e251394d16e390b9d46d208c0b3aa715200f9b28ff25fc6855be07bc1f: Status 404 returned error can't find the container with id 9d8522e251394d16e390b9d46d208c0b3aa715200f9b28ff25fc6855be07bc1f
Jul 24 12:27:34.306113 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:34.306069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b674788f-b72pf" event={"ID":"e6c816c7-b50d-4828-924a-a0928bde88fd","Type":"ContainerStarted","Data":"cc44687b6846745e37b93be812b9a5aedb10ba35d366c0abc461121eda07cef0"}
Jul 24 12:27:34.308459 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:34.308399    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc9bbb694-k48m6" event={"ID":"6b4aa5c5-13d2-4e20-883d-1d87baffc373","Type":"ContainerStarted","Data":"9d8522e251394d16e390b9d46d208c0b3aa715200f9b28ff25fc6855be07bc1f"}
Jul 24 12:27:36.053609 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:36.053572    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:36.153951 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:36.153866    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:27:36.325086 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:36.325049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b674788f-b72pf" event={"ID":"e6c816c7-b50d-4828-924a-a0928bde88fd","Type":"ContainerStarted","Data":"816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde"}
Jul 24 12:27:36.325266 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:36.325126    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:36.326563 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:36.326542    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc9bbb694-k48m6" event={"ID":"6b4aa5c5-13d2-4e20-883d-1d87baffc373","Type":"ContainerStarted","Data":"6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312"}
Jul 24 12:27:36.326682 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:36.326667    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:27:36.350909 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:36.350860    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67b674788f-b72pf" podStartSLOduration=1.88300076 podStartE2EDuration="4.350842876s" totalImagesPullingTime="2.467842116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:27:36.349957869 +0000 UTC m=+529.877704923" watchObservedRunningTime="2026-07-24 12:27:36.350842876 +0000 UTC m=+529.878589930"
Jul 24 12:27:36.379921 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:36.379860    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cc9bbb694-k48m6" podStartSLOduration=2.046366132 podStartE2EDuration="4.379842013s" totalImagesPullingTime="2.333475881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:27:36.379310408 +0000 UTC m=+529.907057471" watchObservedRunningTime="2026-07-24 12:27:36.379842013 +0000 UTC m=+529.907589067"
Jul 24 12:27:47.333729 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:47.333699    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:27:47.334183 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:47.333779    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:47.405853 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:47.405820    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67b674788f-b72pf"]
Jul 24 12:27:47.406053 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:47.406005    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-67b674788f-b72pf" podUID="e6c816c7-b50d-4828-924a-a0928bde88fd" containerName="manager" containerID="cri-o://816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde" gracePeriod=30
Jul 24 12:27:47.658166 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:47.658142    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:47.688881 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:47.688852    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrdmk\" (UniqueName: \"kubernetes.io/projected/e6c816c7-b50d-4828-924a-a0928bde88fd-kube-api-access-xrdmk\") pod \"e6c816c7-b50d-4828-924a-a0928bde88fd\" (UID: \"e6c816c7-b50d-4828-924a-a0928bde88fd\") "
Jul 24 12:27:47.689069 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:47.688930    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e6c816c7-b50d-4828-924a-a0928bde88fd-webhook-cert\") pod \"e6c816c7-b50d-4828-924a-a0928bde88fd\" (UID: \"e6c816c7-b50d-4828-924a-a0928bde88fd\") "
Jul 24 12:27:47.691243 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:47.691207    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c816c7-b50d-4828-924a-a0928bde88fd-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e6c816c7-b50d-4828-924a-a0928bde88fd" (UID: "e6c816c7-b50d-4828-924a-a0928bde88fd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:27:47.691377 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:47.691248    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c816c7-b50d-4828-924a-a0928bde88fd-kube-api-access-xrdmk" (OuterVolumeSpecName: "kube-api-access-xrdmk") pod "e6c816c7-b50d-4828-924a-a0928bde88fd" (UID: "e6c816c7-b50d-4828-924a-a0928bde88fd"). InnerVolumeSpecName "kube-api-access-xrdmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:27:47.789502 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:47.789466    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e6c816c7-b50d-4828-924a-a0928bde88fd-webhook-cert\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:27:47.789502 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:47.789494    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrdmk\" (UniqueName: \"kubernetes.io/projected/e6c816c7-b50d-4828-924a-a0928bde88fd-kube-api-access-xrdmk\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:27:48.385278 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:48.385242    2566 generic.go:356] "Generic (PLEG): container finished" podID="e6c816c7-b50d-4828-924a-a0928bde88fd" containerID="816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde" exitCode=0
Jul 24 12:27:48.385746 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:48.385284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b674788f-b72pf" event={"ID":"e6c816c7-b50d-4828-924a-a0928bde88fd","Type":"ContainerDied","Data":"816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde"}
Jul 24 12:27:48.385746 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:48.385291    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b674788f-b72pf"
Jul 24 12:27:48.385746 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:48.385336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b674788f-b72pf" event={"ID":"e6c816c7-b50d-4828-924a-a0928bde88fd","Type":"ContainerDied","Data":"cc44687b6846745e37b93be812b9a5aedb10ba35d366c0abc461121eda07cef0"}
Jul 24 12:27:48.385746 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:48.385367    2566 scope.go:117] "RemoveContainer" containerID="816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde"
Jul 24 12:27:48.394227 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:48.394209    2566 scope.go:117] "RemoveContainer" containerID="816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde"
Jul 24 12:27:48.394467 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:27:48.394449    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde\": container with ID starting with 816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde not found: ID does not exist" containerID="816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde"
Jul 24 12:27:48.394511 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:48.394476    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde"} err="failed to get container status \"816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde\": rpc error: code = NotFound desc = could not find container \"816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde\": container with ID starting with 816e837c0b0a9962a29a53bcfed25337bb5eceade6cf36e28e06b8469ff6cbde not found: ID does not exist"
Jul 24 12:27:48.406755 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:48.406730    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67b674788f-b72pf"]
Jul 24 12:27:48.410630 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:48.410612    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67b674788f-b72pf"]
Jul 24 12:27:48.975224 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:27:48.975195    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c816c7-b50d-4828-924a-a0928bde88fd" path="/var/lib/kubelet/pods/e6c816c7-b50d-4828-924a-a0928bde88fd/volumes"
Jul 24 12:28:07.780041 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:07.779979    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:07.866726 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:07.866685    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:08.355298 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.355264    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:08.455178 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.455132    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"]
Jul 24 12:28:08.455526 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.455507    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c816c7-b50d-4828-924a-a0928bde88fd" containerName="manager"
Jul 24 12:28:08.455526 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.455529    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c816c7-b50d-4828-924a-a0928bde88fd" containerName="manager"
Jul 24 12:28:08.455694 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.455607    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c816c7-b50d-4828-924a-a0928bde88fd" containerName="manager"
Jul 24 12:28:08.459970 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.459949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:08.464212 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.464186    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 24 12:28:08.464212 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.464192    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plh29\""
Jul 24 12:28:08.474316 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.474293    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"]
Jul 24 12:28:08.516428 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.516396    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:08.563736 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.563701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjvq6\" (UniqueName: \"kubernetes.io/projected/95c7a827-89e9-4a4b-9227-bd72614db1a8-kube-api-access-zjvq6\") pod \"maas-api-76d9db859c-qnhrd\" (UID: \"95c7a827-89e9-4a4b-9227-bd72614db1a8\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:08.563888 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.563745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c7a827-89e9-4a4b-9227-bd72614db1a8-maas-api-tls\") pod \"maas-api-76d9db859c-qnhrd\" (UID: \"95c7a827-89e9-4a4b-9227-bd72614db1a8\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:08.563888 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.563831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c7a827-89e9-4a4b-9227-bd72614db1a8-openshift-service-ca\") pod \"maas-api-76d9db859c-qnhrd\" (UID: \"95c7a827-89e9-4a4b-9227-bd72614db1a8\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:08.664638 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.664550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c7a827-89e9-4a4b-9227-bd72614db1a8-maas-api-tls\") pod \"maas-api-76d9db859c-qnhrd\" (UID: \"95c7a827-89e9-4a4b-9227-bd72614db1a8\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:08.664638 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.664602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c7a827-89e9-4a4b-9227-bd72614db1a8-openshift-service-ca\") pod \"maas-api-76d9db859c-qnhrd\" (UID: \"95c7a827-89e9-4a4b-9227-bd72614db1a8\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:08.664849 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.664655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjvq6\" (UniqueName: \"kubernetes.io/projected/95c7a827-89e9-4a4b-9227-bd72614db1a8-kube-api-access-zjvq6\") pod \"maas-api-76d9db859c-qnhrd\" (UID: \"95c7a827-89e9-4a4b-9227-bd72614db1a8\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:08.665292 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.665264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c7a827-89e9-4a4b-9227-bd72614db1a8-openshift-service-ca\") pod \"maas-api-76d9db859c-qnhrd\" (UID: \"95c7a827-89e9-4a4b-9227-bd72614db1a8\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:08.667193 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.667160    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c7a827-89e9-4a4b-9227-bd72614db1a8-maas-api-tls\") pod \"maas-api-76d9db859c-qnhrd\" (UID: \"95c7a827-89e9-4a4b-9227-bd72614db1a8\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:08.676591 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.676568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjvq6\" (UniqueName: \"kubernetes.io/projected/95c7a827-89e9-4a4b-9227-bd72614db1a8-kube-api-access-zjvq6\") pod \"maas-api-76d9db859c-qnhrd\" (UID: \"95c7a827-89e9-4a4b-9227-bd72614db1a8\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:08.769896 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.769845    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:08.923177 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.923151    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"]
Jul 24 12:28:08.925636 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:28:08.925610    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c7a827_89e9_4a4b_9227_bd72614db1a8.slice/crio-b9b4c8c3109860b1b6aeaa971044332cb52976379d28b54e30467ae2e2b57f69 WatchSource:0}: Error finding container b9b4c8c3109860b1b6aeaa971044332cb52976379d28b54e30467ae2e2b57f69: Status 404 returned error can't find the container with id b9b4c8c3109860b1b6aeaa971044332cb52976379d28b54e30467ae2e2b57f69
Jul 24 12:28:08.986474 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.986445    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-c6knq"]
Jul 24 12:28:08.990927 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.990906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq"
Jul 24 12:28:08.996795 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.996775    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zzndz\""
Jul 24 12:28:08.998102 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:08.998083    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 24 12:28:09.013160 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.013134    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-c6knq"]
Jul 24 12:28:09.015786 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.015760    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"]
Jul 24 12:28:09.019626 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.019605    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"
Jul 24 12:28:09.042198 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.042168    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"]
Jul 24 12:28:09.068613 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.068581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3caa1c0e-87cd-4d2b-99f3-06508b01508a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-mcvc9\" (UID: \"3caa1c0e-87cd-4d2b-99f3-06508b01508a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"
Jul 24 12:28:09.068781 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.068624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca77e87f-a898-4c03-8b7b-ca53cacbf8f7-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-c6knq\" (UID: \"ca77e87f-a898-4c03-8b7b-ca53cacbf8f7\") " pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq"
Jul 24 12:28:09.068781 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.068717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k69m\" (UniqueName: \"kubernetes.io/projected/3caa1c0e-87cd-4d2b-99f3-06508b01508a-kube-api-access-4k69m\") pod \"payload-pre-processing-687dbb6cf6-mcvc9\" (UID: \"3caa1c0e-87cd-4d2b-99f3-06508b01508a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"
Jul 24 12:28:09.068781 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.068771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97zj\" (UniqueName: \"kubernetes.io/projected/ca77e87f-a898-4c03-8b7b-ca53cacbf8f7-kube-api-access-b97zj\") pod \"payload-processing-5cc6cd444b-c6knq\" (UID: \"ca77e87f-a898-4c03-8b7b-ca53cacbf8f7\") " pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq"
Jul 24 12:28:09.170253 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.170208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3caa1c0e-87cd-4d2b-99f3-06508b01508a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-mcvc9\" (UID: \"3caa1c0e-87cd-4d2b-99f3-06508b01508a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"
Jul 24 12:28:09.170253 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.170245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca77e87f-a898-4c03-8b7b-ca53cacbf8f7-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-c6knq\" (UID: \"ca77e87f-a898-4c03-8b7b-ca53cacbf8f7\") " pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq"
Jul 24 12:28:09.170457 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.170384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k69m\" (UniqueName: \"kubernetes.io/projected/3caa1c0e-87cd-4d2b-99f3-06508b01508a-kube-api-access-4k69m\") pod \"payload-pre-processing-687dbb6cf6-mcvc9\" (UID: \"3caa1c0e-87cd-4d2b-99f3-06508b01508a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"
Jul 24 12:28:09.170457 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.170448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97zj\" (UniqueName: \"kubernetes.io/projected/ca77e87f-a898-4c03-8b7b-ca53cacbf8f7-kube-api-access-b97zj\") pod \"payload-processing-5cc6cd444b-c6knq\" (UID: \"ca77e87f-a898-4c03-8b7b-ca53cacbf8f7\") " pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq"
Jul 24 12:28:09.170981 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.170956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca77e87f-a898-4c03-8b7b-ca53cacbf8f7-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-c6knq\" (UID: \"ca77e87f-a898-4c03-8b7b-ca53cacbf8f7\") " pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq"
Jul 24 12:28:09.171124 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.170992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3caa1c0e-87cd-4d2b-99f3-06508b01508a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-mcvc9\" (UID: \"3caa1c0e-87cd-4d2b-99f3-06508b01508a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"
Jul 24 12:28:09.179355 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.179298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k69m\" (UniqueName: \"kubernetes.io/projected/3caa1c0e-87cd-4d2b-99f3-06508b01508a-kube-api-access-4k69m\") pod \"payload-pre-processing-687dbb6cf6-mcvc9\" (UID: \"3caa1c0e-87cd-4d2b-99f3-06508b01508a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"
Jul 24 12:28:09.179355 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.179338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97zj\" (UniqueName: \"kubernetes.io/projected/ca77e87f-a898-4c03-8b7b-ca53cacbf8f7-kube-api-access-b97zj\") pod \"payload-processing-5cc6cd444b-c6knq\" (UID: \"ca77e87f-a898-4c03-8b7b-ca53cacbf8f7\") " pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq"
Jul 24 12:28:09.302503 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.302461    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq"
Jul 24 12:28:09.330795 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.330761    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"
Jul 24 12:28:09.355714 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.355659    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:09.453680 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.453636    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:09.459780 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.459752    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-c6knq"]
Jul 24 12:28:09.460638 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:28:09.460599    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca77e87f_a898_4c03_8b7b_ca53cacbf8f7.slice/crio-80d4ee07ffef89f9ebac94c20ba855432e5a0b02e6964f02b3894b08bbd9ba39 WatchSource:0}: Error finding container 80d4ee07ffef89f9ebac94c20ba855432e5a0b02e6964f02b3894b08bbd9ba39: Status 404 returned error can't find the container with id 80d4ee07ffef89f9ebac94c20ba855432e5a0b02e6964f02b3894b08bbd9ba39
Jul 24 12:28:09.486352 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.486316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq" event={"ID":"ca77e87f-a898-4c03-8b7b-ca53cacbf8f7","Type":"ContainerStarted","Data":"80d4ee07ffef89f9ebac94c20ba855432e5a0b02e6964f02b3894b08bbd9ba39"}
Jul 24 12:28:09.487629 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.487604    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd" event={"ID":"95c7a827-89e9-4a4b-9227-bd72614db1a8","Type":"ContainerStarted","Data":"b9b4c8c3109860b1b6aeaa971044332cb52976379d28b54e30467ae2e2b57f69"}
Jul 24 12:28:09.487828 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:09.487732    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"]
Jul 24 12:28:09.490034 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:28:09.489992    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3caa1c0e_87cd_4d2b_99f3_06508b01508a.slice/crio-53a48a9904c65877f60349304325793913ee0940c3102863337afe9aae6601de WatchSource:0}: Error finding container 53a48a9904c65877f60349304325793913ee0940c3102863337afe9aae6601de: Status 404 returned error can't find the container with id 53a48a9904c65877f60349304325793913ee0940c3102863337afe9aae6601de
Jul 24 12:28:10.495702 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:10.495626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9" event={"ID":"3caa1c0e-87cd-4d2b-99f3-06508b01508a","Type":"ContainerStarted","Data":"53a48a9904c65877f60349304325793913ee0940c3102863337afe9aae6601de"}
Jul 24 12:28:10.922977 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:10.922893    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:13.513898 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:13.513863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9" event={"ID":"3caa1c0e-87cd-4d2b-99f3-06508b01508a","Type":"ContainerStarted","Data":"d71f6c6dc5cd1df6a634eaab113c9f19789b64c2f3548bcd86e367b941d06c97"}
Jul 24 12:28:13.514368 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:13.513940    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"
Jul 24 12:28:13.515636 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:13.515583    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq" event={"ID":"ca77e87f-a898-4c03-8b7b-ca53cacbf8f7","Type":"ContainerStarted","Data":"e50afa1eb50f6c9ff9fc35ec7ef206c2cccca9ccf96167a8cbe26341c264bc50"}
Jul 24 12:28:13.515790 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:13.515657    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq"
Jul 24 12:28:13.517182 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:13.517162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd" event={"ID":"95c7a827-89e9-4a4b-9227-bd72614db1a8","Type":"ContainerStarted","Data":"5fcc5ec24b8f225f8983a8c935e7b45ea23f73d9ce3a2b77eeeebf46c22ef010"}
Jul 24 12:28:13.517317 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:13.517300    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:13.532463 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:13.532419    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9" podStartSLOduration=1.845708114 podStartE2EDuration="5.532403773s" totalImagesPullingTime="3.686695659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:28:13.531010938 +0000 UTC m=+567.058757992" watchObservedRunningTime="2026-07-24 12:28:13.532403773 +0000 UTC m=+567.060150827"
Jul 24 12:28:13.582191 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:13.582138    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd" podStartSLOduration=1.3366093079999999 podStartE2EDuration="5.582120352s" totalImagesPullingTime="4.245511044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:28:13.566851594 +0000 UTC m=+567.094598649" watchObservedRunningTime="2026-07-24 12:28:13.582120352 +0000 UTC m=+567.109867410"
Jul 24 12:28:13.592376 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:13.592324    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq" podStartSLOduration=1.882399352 podStartE2EDuration="5.59230919s" totalImagesPullingTime="3.709909838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:28:13.591989563 +0000 UTC m=+567.119736618" watchObservedRunningTime="2026-07-24 12:28:13.59230919 +0000 UTC m=+567.120056245"
Jul 24 12:28:19.528131 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:19.528097    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76d9db859c-qnhrd"
Jul 24 12:28:20.567429 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:20.567383    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:20.661754 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:20.661712    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:24.523884 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:24.523850    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mcvc9"
Jul 24 12:28:24.524372 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:24.524158    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-c6knq"
Jul 24 12:28:26.555616 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:26.555575    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:34.318344 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.318299    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"]
Jul 24 12:28:34.323135 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.323110    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.328069 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.328046    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 12:28:34.328196 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.328080    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6spvv\""
Jul 24 12:28:34.328196 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.328045    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 24 12:28:34.328196 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.328085    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 12:28:34.343576 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.343553    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"]
Jul 24 12:28:34.406596 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.406557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh7dp\" (UniqueName: \"kubernetes.io/projected/848593e9-9746-4e5e-95a7-ce1f8e99d18b-kube-api-access-gh7dp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.406757 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.406613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848593e9-9746-4e5e-95a7-ce1f8e99d18b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.406757 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.406674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848593e9-9746-4e5e-95a7-ce1f8e99d18b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.406757 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.406701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848593e9-9746-4e5e-95a7-ce1f8e99d18b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.406757 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.406753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848593e9-9746-4e5e-95a7-ce1f8e99d18b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.406888 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.406798    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848593e9-9746-4e5e-95a7-ce1f8e99d18b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.428787 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.428757    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:34.508182 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.508144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848593e9-9746-4e5e-95a7-ce1f8e99d18b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.508370 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.508188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848593e9-9746-4e5e-95a7-ce1f8e99d18b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.508370 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.508218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh7dp\" (UniqueName: \"kubernetes.io/projected/848593e9-9746-4e5e-95a7-ce1f8e99d18b-kube-api-access-gh7dp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.508370 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.508282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848593e9-9746-4e5e-95a7-ce1f8e99d18b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.508370 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.508313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848593e9-9746-4e5e-95a7-ce1f8e99d18b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.508578 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.508495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848593e9-9746-4e5e-95a7-ce1f8e99d18b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.508633 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.508576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848593e9-9746-4e5e-95a7-ce1f8e99d18b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.508728 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.508709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848593e9-9746-4e5e-95a7-ce1f8e99d18b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.508899 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.508875    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848593e9-9746-4e5e-95a7-ce1f8e99d18b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.510654 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.510631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848593e9-9746-4e5e-95a7-ce1f8e99d18b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.510921 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.510901    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848593e9-9746-4e5e-95a7-ce1f8e99d18b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.517126 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.517105    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh7dp\" (UniqueName: \"kubernetes.io/projected/848593e9-9746-4e5e-95a7-ce1f8e99d18b-kube-api-access-gh7dp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9khb2\" (UID: \"848593e9-9746-4e5e-95a7-ce1f8e99d18b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.634395 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.634320    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:34.770867 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:34.770833    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"]
Jul 24 12:28:34.772667 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:28:34.772644    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848593e9_9746_4e5e_95a7_ce1f8e99d18b.slice/crio-70123fe523271dbbc1f4b43af94ae99e5f29a05d27d79a8632ad88fa17065651 WatchSource:0}: Error finding container 70123fe523271dbbc1f4b43af94ae99e5f29a05d27d79a8632ad88fa17065651: Status 404 returned error can't find the container with id 70123fe523271dbbc1f4b43af94ae99e5f29a05d27d79a8632ad88fa17065651
Jul 24 12:28:35.621843 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:35.621813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2" event={"ID":"848593e9-9746-4e5e-95a7-ce1f8e99d18b","Type":"ContainerStarted","Data":"70123fe523271dbbc1f4b43af94ae99e5f29a05d27d79a8632ad88fa17065651"}
Jul 24 12:28:36.628790 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:36.628753    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2" event={"ID":"848593e9-9746-4e5e-95a7-ce1f8e99d18b","Type":"ContainerStarted","Data":"bac412b7239d12905fba5661d66db29d0f7b5351d794b5b103dd8654e0201565"}
Jul 24 12:28:36.648929 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:36.648877    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2" podStartSLOduration=1.61726641 podStartE2EDuration="2.648857673s" totalImagesPullingTime="1.031591263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:28:36.646894157 +0000 UTC m=+590.174641211" watchObservedRunningTime="2026-07-24 12:28:36.648857673 +0000 UTC m=+590.176604726"
Jul 24 12:28:44.634893 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:44.634845    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:44.635545 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:44.634967    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:44.647749 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:44.647715    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:44.679546 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:44.679517    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9khb2"
Jul 24 12:28:46.929224 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:46.929199    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:28:46.932742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:46.932714    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:28:48.164098 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:48.164056    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:54.419189 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.419156    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"]
Jul 24 12:28:54.422220 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.422193    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.426089 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.425737    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 24 12:28:54.439056 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.439032    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"]
Jul 24 12:28:54.460674 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.460641    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:28:54.491439 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.491411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.491590 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.491447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.491590 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.491482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.491687 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.491581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxkg\" (UniqueName: \"kubernetes.io/projected/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-kube-api-access-xsxkg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.491687 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.491630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.491687 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.491648    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.594880 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.594839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.594880 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.594871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.595245 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.594937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.595245 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.594960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.595245 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.595084    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.595245 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.595172    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxkg\" (UniqueName: \"kubernetes.io/projected/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-kube-api-access-xsxkg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.595479 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.595406    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.595479 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.595434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.595581 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.595513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.597446 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.597418    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.597599 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.597584    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.604455 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.604435    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxkg\" (UniqueName: \"kubernetes.io/projected/7d0f1d38-12ee-4fcf-baf2-4b5fcf462626-kube-api-access-xsxkg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv\" (UID: \"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.733237 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.733146    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:28:54.879968 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.879942    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"]
Jul 24 12:28:54.881001 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:28:54.880975    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0f1d38_12ee_4fcf_baf2_4b5fcf462626.slice/crio-e6c3e3466daf93d4f379fa0d0200586c3fda71aa6ae8822b92ebc9551024b7d3 WatchSource:0}: Error finding container e6c3e3466daf93d4f379fa0d0200586c3fda71aa6ae8822b92ebc9551024b7d3: Status 404 returned error can't find the container with id e6c3e3466daf93d4f379fa0d0200586c3fda71aa6ae8822b92ebc9551024b7d3
Jul 24 12:28:54.882336 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:54.882319    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 12:28:55.726105 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:55.726069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv" event={"ID":"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626","Type":"ContainerStarted","Data":"e0c42116bf27bbd007c7520473f0b65a0bf456f164f4134865aefe0f6edd0ae6"}
Jul 24 12:28:55.726611 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:55.726132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv" event={"ID":"7d0f1d38-12ee-4fcf-baf2-4b5fcf462626","Type":"ContainerStarted","Data":"e6c3e3466daf93d4f379fa0d0200586c3fda71aa6ae8822b92ebc9551024b7d3"}
Jul 24 12:28:55.746558 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:28:55.746509    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv" podStartSLOduration=1.519990733 podStartE2EDuration="1.746493701s" totalImagesPullingTime="226.502968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:28:55.745321672 +0000 UTC m=+609.273068727" watchObservedRunningTime="2026-07-24 12:28:55.746493701 +0000 UTC m=+609.274240796"
Jul 24 12:29:01.928118 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:01.928088    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"]
Jul 24 12:29:01.930591 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:01.930572    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:01.933842 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:01.933817    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 24 12:29:01.940801 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:01.940777    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"]
Jul 24 12:29:01.957660 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:01.957632    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:29:02.060473 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.060443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.060639 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.060486    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.060639 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.060517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.060639 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.060600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmhd\" (UniqueName: \"kubernetes.io/projected/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-kube-api-access-7mmhd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.060742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.060666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.060742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.060703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.076397 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.076369    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:29:02.162217 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.162180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.162383 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.162243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.162383 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.162281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.162383 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.162321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmhd\" (UniqueName: \"kubernetes.io/projected/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-kube-api-access-7mmhd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.162383 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.162375    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.162558 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.162409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.162734 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.162677    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.162843 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.162782    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.163051 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.162979    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.164694 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.164673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.165187 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.165169    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.183430 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.183367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmhd\" (UniqueName: \"kubernetes.io/projected/a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1-kube-api-access-7mmhd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g7894\" (UID: \"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.242257 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.242225    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:02.381097 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.381074    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"]
Jul 24 12:29:02.382617 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:29:02.382595    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15675f7_a7d5_43c1_a1ba_ce8eeb0dc6c1.slice/crio-fcaa961424352fca6bda379e216802914c0c95672738c18d60dd1f6f990d1bda WatchSource:0}: Error finding container fcaa961424352fca6bda379e216802914c0c95672738c18d60dd1f6f990d1bda: Status 404 returned error can't find the container with id fcaa961424352fca6bda379e216802914c0c95672738c18d60dd1f6f990d1bda
Jul 24 12:29:02.759501 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.759470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894" event={"ID":"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1","Type":"ContainerStarted","Data":"59cac86d95aa96fc2ca3a820abaaf799bc7fdcf43e58e7be54b6ef659a5ef533"}
Jul 24 12:29:02.759673 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.759524    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894" event={"ID":"a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1","Type":"ContainerStarted","Data":"fcaa961424352fca6bda379e216802914c0c95672738c18d60dd1f6f990d1bda"}
Jul 24 12:29:02.781511 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:02.781459    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894" podStartSLOduration=1.533066386 podStartE2EDuration="1.781443895s" totalImagesPullingTime="248.377509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:29:02.779906363 +0000 UTC m=+616.307653430" watchObservedRunningTime="2026-07-24 12:29:02.781443895 +0000 UTC m=+616.309190948"
Jul 24 12:29:04.734166 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:04.734138    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:29:04.734549 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:04.734393    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:29:04.746939 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:04.746908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:29:04.782681 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:04.782657    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv"
Jul 24 12:29:12.242940 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:12.242865    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:12.242940 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:12.242924    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:12.255417 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:12.255393    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:12.820090 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:12.820064    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g7894"
Jul 24 12:29:21.253265 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:21.253230    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:29:30.560655 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:30.560604    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:29:30.624370 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:29:30.624332    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:30:02.267332 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:30:02.267286    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:30:02.643738 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:30:02.643630    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:30:12.050408 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:30:12.050376    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:30:12.141843 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:30:12.141816    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:30:13.312428 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:30:13.312390    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:30:13.416050 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:30:13.415984    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:31:01.529373 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:01.529333    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cc9bbb694-k48m6"]
Jul 24 12:31:01.529916 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:01.529527    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cc9bbb694-k48m6" podUID="6b4aa5c5-13d2-4e20-883d-1d87baffc373" containerName="manager" containerID="cri-o://6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312" gracePeriod=30
Jul 24 12:31:01.786967 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:01.786913    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:31:01.843972 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:01.843939    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcckv\" (UniqueName: \"kubernetes.io/projected/6b4aa5c5-13d2-4e20-883d-1d87baffc373-kube-api-access-tcckv\") pod \"6b4aa5c5-13d2-4e20-883d-1d87baffc373\" (UID: \"6b4aa5c5-13d2-4e20-883d-1d87baffc373\") "
Jul 24 12:31:01.844170 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:01.843993    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b4aa5c5-13d2-4e20-883d-1d87baffc373-webhook-cert\") pod \"6b4aa5c5-13d2-4e20-883d-1d87baffc373\" (UID: \"6b4aa5c5-13d2-4e20-883d-1d87baffc373\") "
Jul 24 12:31:01.846134 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:01.846104    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4aa5c5-13d2-4e20-883d-1d87baffc373-kube-api-access-tcckv" (OuterVolumeSpecName: "kube-api-access-tcckv") pod "6b4aa5c5-13d2-4e20-883d-1d87baffc373" (UID: "6b4aa5c5-13d2-4e20-883d-1d87baffc373"). InnerVolumeSpecName "kube-api-access-tcckv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:31:01.846279 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:01.846176    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4aa5c5-13d2-4e20-883d-1d87baffc373-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6b4aa5c5-13d2-4e20-883d-1d87baffc373" (UID: "6b4aa5c5-13d2-4e20-883d-1d87baffc373"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:31:01.951198 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:01.951156    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcckv\" (UniqueName: \"kubernetes.io/projected/6b4aa5c5-13d2-4e20-883d-1d87baffc373-kube-api-access-tcckv\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:31:01.951480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:01.951459    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b4aa5c5-13d2-4e20-883d-1d87baffc373-webhook-cert\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:31:02.336816 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:02.336780    2566 generic.go:356] "Generic (PLEG): container finished" podID="6b4aa5c5-13d2-4e20-883d-1d87baffc373" containerID="6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312" exitCode=0
Jul 24 12:31:02.337009 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:02.336849    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc9bbb694-k48m6"
Jul 24 12:31:02.337009 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:02.336872    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc9bbb694-k48m6" event={"ID":"6b4aa5c5-13d2-4e20-883d-1d87baffc373","Type":"ContainerDied","Data":"6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312"}
Jul 24 12:31:02.337009 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:02.336928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc9bbb694-k48m6" event={"ID":"6b4aa5c5-13d2-4e20-883d-1d87baffc373","Type":"ContainerDied","Data":"9d8522e251394d16e390b9d46d208c0b3aa715200f9b28ff25fc6855be07bc1f"}
Jul 24 12:31:02.337009 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:02.336950    2566 scope.go:117] "RemoveContainer" containerID="6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312"
Jul 24 12:31:02.346318 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:02.346301    2566 scope.go:117] "RemoveContainer" containerID="6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312"
Jul 24 12:31:02.346572 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:31:02.346554    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312\": container with ID starting with 6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312 not found: ID does not exist" containerID="6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312"
Jul 24 12:31:02.346629 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:02.346585    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312"} err="failed to get container status \"6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312\": rpc error: code = NotFound desc = could not find container \"6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312\": container with ID starting with 6aa50dac10869301336a237b2cd0441af659fe3c42c88ae1ac28540e6e4f6312 not found: ID does not exist"
Jul 24 12:31:02.366098 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:02.366073    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cc9bbb694-k48m6"]
Jul 24 12:31:02.378339 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:02.378317    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cc9bbb694-k48m6"]
Jul 24 12:31:02.975400 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:02.975367    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b4aa5c5-13d2-4e20-883d-1d87baffc373" path="/var/lib/kubelet/pods/6b4aa5c5-13d2-4e20-883d-1d87baffc373/volumes"
Jul 24 12:31:53.819213 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:31:53.819179    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:32:03.214705 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.214674    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cc9bbb694-f7hg8"]
Jul 24 12:32:03.215223 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.215076    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b4aa5c5-13d2-4e20-883d-1d87baffc373" containerName="manager"
Jul 24 12:32:03.215223 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.215088    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b4aa5c5-13d2-4e20-883d-1d87baffc373" containerName="manager"
Jul 24 12:32:03.215223 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.215152    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b4aa5c5-13d2-4e20-883d-1d87baffc373" containerName="manager"
Jul 24 12:32:03.219394 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.219377    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc9bbb694-f7hg8"
Jul 24 12:32:03.222510 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.222489    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 12:32:03.222597 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.222531    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 12:32:03.223889 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.223876    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lb9qp\""
Jul 24 12:32:03.230477 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.230451    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc9bbb694-f7hg8"]
Jul 24 12:32:03.318446 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.318411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40e5e19c-e602-455c-a649-7bc2a35356a9-webhook-cert\") pod \"maas-controller-cc9bbb694-f7hg8\" (UID: \"40e5e19c-e602-455c-a649-7bc2a35356a9\") " pod="opendatahub/maas-controller-cc9bbb694-f7hg8"
Jul 24 12:32:03.318629 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.318457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78q5\" (UniqueName: \"kubernetes.io/projected/40e5e19c-e602-455c-a649-7bc2a35356a9-kube-api-access-h78q5\") pod \"maas-controller-cc9bbb694-f7hg8\" (UID: \"40e5e19c-e602-455c-a649-7bc2a35356a9\") " pod="opendatahub/maas-controller-cc9bbb694-f7hg8"
Jul 24 12:32:03.419193 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.419153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40e5e19c-e602-455c-a649-7bc2a35356a9-webhook-cert\") pod \"maas-controller-cc9bbb694-f7hg8\" (UID: \"40e5e19c-e602-455c-a649-7bc2a35356a9\") " pod="opendatahub/maas-controller-cc9bbb694-f7hg8"
Jul 24 12:32:03.419193 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.419200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78q5\" (UniqueName: \"kubernetes.io/projected/40e5e19c-e602-455c-a649-7bc2a35356a9-kube-api-access-h78q5\") pod \"maas-controller-cc9bbb694-f7hg8\" (UID: \"40e5e19c-e602-455c-a649-7bc2a35356a9\") " pod="opendatahub/maas-controller-cc9bbb694-f7hg8"
Jul 24 12:32:03.421789 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.421760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40e5e19c-e602-455c-a649-7bc2a35356a9-webhook-cert\") pod \"maas-controller-cc9bbb694-f7hg8\" (UID: \"40e5e19c-e602-455c-a649-7bc2a35356a9\") " pod="opendatahub/maas-controller-cc9bbb694-f7hg8"
Jul 24 12:32:03.428960 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.428932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78q5\" (UniqueName: \"kubernetes.io/projected/40e5e19c-e602-455c-a649-7bc2a35356a9-kube-api-access-h78q5\") pod \"maas-controller-cc9bbb694-f7hg8\" (UID: \"40e5e19c-e602-455c-a649-7bc2a35356a9\") " pod="opendatahub/maas-controller-cc9bbb694-f7hg8"
Jul 24 12:32:03.531557 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.531526    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc9bbb694-f7hg8"
Jul 24 12:32:03.677612 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:03.677578    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc9bbb694-f7hg8"]
Jul 24 12:32:03.680845 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:32:03.680820    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e5e19c_e602_455c_a649_7bc2a35356a9.slice/crio-d5660933b76742de22f0eff573ef4a7d8f027bccdb750c3df7a85e7695945f8f WatchSource:0}: Error finding container d5660933b76742de22f0eff573ef4a7d8f027bccdb750c3df7a85e7695945f8f: Status 404 returned error can't find the container with id d5660933b76742de22f0eff573ef4a7d8f027bccdb750c3df7a85e7695945f8f
Jul 24 12:32:04.641913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:04.641867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc9bbb694-f7hg8" event={"ID":"40e5e19c-e602-455c-a649-7bc2a35356a9","Type":"ContainerStarted","Data":"5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c"}
Jul 24 12:32:04.642408 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:04.641937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc9bbb694-f7hg8" event={"ID":"40e5e19c-e602-455c-a649-7bc2a35356a9","Type":"ContainerStarted","Data":"d5660933b76742de22f0eff573ef4a7d8f027bccdb750c3df7a85e7695945f8f"}
Jul 24 12:32:04.642408 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:04.641964    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cc9bbb694-f7hg8"
Jul 24 12:32:04.662520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:04.662466    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cc9bbb694-f7hg8" podStartSLOduration=1.207003484 podStartE2EDuration="1.662449784s" totalImagesPullingTime="455.4463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:32:04.659814692 +0000 UTC m=+798.187561746" watchObservedRunningTime="2026-07-24 12:32:04.662449784 +0000 UTC m=+798.190196837"
Jul 24 12:32:15.649334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:15.649251    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cc9bbb694-f7hg8"
Jul 24 12:32:20.206105 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.206070    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:32:20.316709 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.316671    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cc9bbb694-f7hg8"]
Jul 24 12:32:20.316928 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.316902    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cc9bbb694-f7hg8" podUID="40e5e19c-e602-455c-a649-7bc2a35356a9" containerName="manager" containerID="cri-o://5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c" gracePeriod=30
Jul 24 12:32:20.567509 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.567483    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc9bbb694-f7hg8"
Jul 24 12:32:20.643699 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.643670    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h78q5\" (UniqueName: \"kubernetes.io/projected/40e5e19c-e602-455c-a649-7bc2a35356a9-kube-api-access-h78q5\") pod \"40e5e19c-e602-455c-a649-7bc2a35356a9\" (UID: \"40e5e19c-e602-455c-a649-7bc2a35356a9\") "
Jul 24 12:32:20.643904 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.643734    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40e5e19c-e602-455c-a649-7bc2a35356a9-webhook-cert\") pod \"40e5e19c-e602-455c-a649-7bc2a35356a9\" (UID: \"40e5e19c-e602-455c-a649-7bc2a35356a9\") "
Jul 24 12:32:20.645945 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.645916    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e5e19c-e602-455c-a649-7bc2a35356a9-kube-api-access-h78q5" (OuterVolumeSpecName: "kube-api-access-h78q5") pod "40e5e19c-e602-455c-a649-7bc2a35356a9" (UID: "40e5e19c-e602-455c-a649-7bc2a35356a9"). InnerVolumeSpecName "kube-api-access-h78q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:32:20.645945 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.645918    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40e5e19c-e602-455c-a649-7bc2a35356a9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "40e5e19c-e602-455c-a649-7bc2a35356a9" (UID: "40e5e19c-e602-455c-a649-7bc2a35356a9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:32:20.718850 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.718814    2566 generic.go:356] "Generic (PLEG): container finished" podID="40e5e19c-e602-455c-a649-7bc2a35356a9" containerID="5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c" exitCode=0
Jul 24 12:32:20.719065 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.718860    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc9bbb694-f7hg8"
Jul 24 12:32:20.719065 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.718900    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc9bbb694-f7hg8" event={"ID":"40e5e19c-e602-455c-a649-7bc2a35356a9","Type":"ContainerDied","Data":"5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c"}
Jul 24 12:32:20.719065 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.718962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc9bbb694-f7hg8" event={"ID":"40e5e19c-e602-455c-a649-7bc2a35356a9","Type":"ContainerDied","Data":"d5660933b76742de22f0eff573ef4a7d8f027bccdb750c3df7a85e7695945f8f"}
Jul 24 12:32:20.719065 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.718993    2566 scope.go:117] "RemoveContainer" containerID="5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c"
Jul 24 12:32:20.728558 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.728541    2566 scope.go:117] "RemoveContainer" containerID="5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c"
Jul 24 12:32:20.728819 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:32:20.728801    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c\": container with ID starting with 5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c not found: ID does not exist" containerID="5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c"
Jul 24 12:32:20.728858 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.728826    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c"} err="failed to get container status \"5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c\": rpc error: code = NotFound desc = could not find container \"5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c\": container with ID starting with 5bf4df528af275897437203cc4d5ede41bfca4a35d65964e7559a02e2041325c not found: ID does not exist"
Jul 24 12:32:20.740605 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.740553    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cc9bbb694-f7hg8"]
Jul 24 12:32:20.744427 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.744401    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cc9bbb694-f7hg8"]
Jul 24 12:32:20.744771 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.744750    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h78q5\" (UniqueName: \"kubernetes.io/projected/40e5e19c-e602-455c-a649-7bc2a35356a9-kube-api-access-h78q5\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:32:20.744942 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.744777    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40e5e19c-e602-455c-a649-7bc2a35356a9-webhook-cert\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:32:20.974994 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:20.974962    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40e5e19c-e602-455c-a649-7bc2a35356a9" path="/var/lib/kubelet/pods/40e5e19c-e602-455c-a649-7bc2a35356a9/volumes"
Jul 24 12:32:21.454408 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.454377    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:32:21.565113 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.565079    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cc9bbb694-pvqd5"]
Jul 24 12:32:21.565476 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.565463    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e5e19c-e602-455c-a649-7bc2a35356a9" containerName="manager"
Jul 24 12:32:21.565527 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.565480    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e5e19c-e602-455c-a649-7bc2a35356a9" containerName="manager"
Jul 24 12:32:21.565694 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.565667    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e5e19c-e602-455c-a649-7bc2a35356a9" containerName="manager"
Jul 24 12:32:21.568864 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.568847    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc9bbb694-pvqd5"
Jul 24 12:32:21.573345 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.573323    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lb9qp\""
Jul 24 12:32:21.573452 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.573325    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 12:32:21.573452 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.573374    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 12:32:21.578697 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.578674    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc9bbb694-pvqd5"]
Jul 24 12:32:21.653316 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.653283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4b68762-40f8-4cff-a99b-bb94935bcc07-webhook-cert\") pod \"maas-controller-cc9bbb694-pvqd5\" (UID: \"c4b68762-40f8-4cff-a99b-bb94935bcc07\") " pod="opendatahub/maas-controller-cc9bbb694-pvqd5"
Jul 24 12:32:21.653506 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.653371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz88l\" (UniqueName: \"kubernetes.io/projected/c4b68762-40f8-4cff-a99b-bb94935bcc07-kube-api-access-mz88l\") pod \"maas-controller-cc9bbb694-pvqd5\" (UID: \"c4b68762-40f8-4cff-a99b-bb94935bcc07\") " pod="opendatahub/maas-controller-cc9bbb694-pvqd5"
Jul 24 12:32:21.753914 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.753880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz88l\" (UniqueName: \"kubernetes.io/projected/c4b68762-40f8-4cff-a99b-bb94935bcc07-kube-api-access-mz88l\") pod \"maas-controller-cc9bbb694-pvqd5\" (UID: \"c4b68762-40f8-4cff-a99b-bb94935bcc07\") " pod="opendatahub/maas-controller-cc9bbb694-pvqd5"
Jul 24 12:32:21.754122 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.753963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4b68762-40f8-4cff-a99b-bb94935bcc07-webhook-cert\") pod \"maas-controller-cc9bbb694-pvqd5\" (UID: \"c4b68762-40f8-4cff-a99b-bb94935bcc07\") " pod="opendatahub/maas-controller-cc9bbb694-pvqd5"
Jul 24 12:32:21.756515 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.756487    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4b68762-40f8-4cff-a99b-bb94935bcc07-webhook-cert\") pod \"maas-controller-cc9bbb694-pvqd5\" (UID: \"c4b68762-40f8-4cff-a99b-bb94935bcc07\") " pod="opendatahub/maas-controller-cc9bbb694-pvqd5"
Jul 24 12:32:21.771135 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.771108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz88l\" (UniqueName: \"kubernetes.io/projected/c4b68762-40f8-4cff-a99b-bb94935bcc07-kube-api-access-mz88l\") pod \"maas-controller-cc9bbb694-pvqd5\" (UID: \"c4b68762-40f8-4cff-a99b-bb94935bcc07\") " pod="opendatahub/maas-controller-cc9bbb694-pvqd5"
Jul 24 12:32:21.880474 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:21.880434    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc9bbb694-pvqd5"
Jul 24 12:32:22.025065 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:22.022121    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc9bbb694-pvqd5"]
Jul 24 12:32:22.028139 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:32:22.028103    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b68762_40f8_4cff_a99b_bb94935bcc07.slice/crio-ea9fab637ab1888a48776debd57d4cd6f004dd99acb53c2543c2f20611769272 WatchSource:0}: Error finding container ea9fab637ab1888a48776debd57d4cd6f004dd99acb53c2543c2f20611769272: Status 404 returned error can't find the container with id ea9fab637ab1888a48776debd57d4cd6f004dd99acb53c2543c2f20611769272
Jul 24 12:32:22.732669 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:22.732583    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc9bbb694-pvqd5" event={"ID":"c4b68762-40f8-4cff-a99b-bb94935bcc07","Type":"ContainerStarted","Data":"5ad4edaccd5a9f8d7801586b5a699f8a5111a2333eb352b9d0c75d49c6ab0bb5"}
Jul 24 12:32:22.732669 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:22.732636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc9bbb694-pvqd5" event={"ID":"c4b68762-40f8-4cff-a99b-bb94935bcc07","Type":"ContainerStarted","Data":"ea9fab637ab1888a48776debd57d4cd6f004dd99acb53c2543c2f20611769272"}
Jul 24 12:32:22.733123 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:22.732705    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cc9bbb694-pvqd5"
Jul 24 12:32:22.754895 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:22.754847    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cc9bbb694-pvqd5" podStartSLOduration=1.449656889 podStartE2EDuration="1.754830655s" totalImagesPullingTime="305.173766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:32:22.752902831 +0000 UTC m=+816.280649885" watchObservedRunningTime="2026-07-24 12:32:22.754830655 +0000 UTC m=+816.282577708"
Jul 24 12:32:33.741286 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:33.741256    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cc9bbb694-pvqd5"
Jul 24 12:32:59.443195 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:59.443153    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:32:59.570565 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:59.570538    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:32:59.671599 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:32:59.671563    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:33:05.476667 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:33:05.476632    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:33:05.581090 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:33:05.581058    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:33:05.676586 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:33:05.676555    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:33:05.843160 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:33:05.843130    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:33:11.896839 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:33:11.896807    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:33:11.965490 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:33:11.965459    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:33:46.959037 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:33:46.958991    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:33:46.960069 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:33:46.960048    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:34:18.080771 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:34:18.080733    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:34:18.282676 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:34:18.282645    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:34:25.592571 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:34:25.592536    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:34:26.882900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:34:26.882866    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:35:19.799743 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:35:19.799670    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:38:10.389860 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:38:10.389769    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:38:46.988150 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:38:46.988120    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:38:46.989370 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:38:46.989351    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:40:48.663784 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:40:48.663750    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:41:08.923465 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:41:08.923371    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:41:45.234954 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:41:45.234920    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:41:46.178692 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:41:46.178658    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:43:16.340608 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:43:16.340569    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:43:47.017005 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:43:47.016974    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:43:47.019404 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:43:47.019379    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:43:52.734435 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:43:52.734397    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:43:57.461278 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:43:57.461243    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:43:57.559084 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:43:57.559010    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:44:21.028352 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:44:21.028263    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:44:58.949718 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:44:58.949682    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:20.595896 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:20.595863    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:23.823223 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:23.823154    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:23.922711 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:23.922669    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:24.290900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:24.290862    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:30.066178 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:30.066141    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:30.159470 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:30.159432    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:30.298837 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:30.298797    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:30.625737 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:30.625697    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:30.872193 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:30.872160    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:32.575765 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.575731    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:32.672788 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.672756    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"]
Jul 24 12:45:32.673372 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.673344    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:32.676504 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.676488    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:32.680877 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.680856    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-227c45ac-serving-cert\""
Jul 24 12:45:32.690553 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.690528    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"]
Jul 24 12:45:32.749838 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.749792    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:32.832580 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.832478    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-maas-api-tls\") pod \"maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9\" (UID: \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:32.832580 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.832519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-openshift-service-ca\") pod \"maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9\" (UID: \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:32.832840 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.832668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkgqm\" (UniqueName: \"kubernetes.io/projected/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-kube-api-access-hkgqm\") pod \"maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9\" (UID: \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:32.909481 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.909445    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:32.933578 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.933542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgqm\" (UniqueName: \"kubernetes.io/projected/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-kube-api-access-hkgqm\") pod \"maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9\" (UID: \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:32.933757 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.933635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-maas-api-tls\") pod \"maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9\" (UID: \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:32.933757 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.933656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-openshift-service-ca\") pod \"maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9\" (UID: \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:32.934642 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.934614    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-openshift-service-ca\") pod \"maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9\" (UID: \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:32.936465 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.936432    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-maas-api-tls\") pod \"maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9\" (UID: \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:32.944879 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.944850    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkgqm\" (UniqueName: \"kubernetes.io/projected/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-kube-api-access-hkgqm\") pod \"maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9\" (UID: \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:32.987006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:32.986967    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:33.132433 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.132403    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"]
Jul 24 12:45:33.134184 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:45:33.134146    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a8731a_d9bc_41cd_9a00_98cc50f0f6a5.slice/crio-58e3001b9dcf8798c0e7c5e2f5841cb60259561cdf47fdb74ed889745297c35d WatchSource:0}: Error finding container 58e3001b9dcf8798c0e7c5e2f5841cb60259561cdf47fdb74ed889745297c35d: Status 404 returned error can't find the container with id 58e3001b9dcf8798c0e7c5e2f5841cb60259561cdf47fdb74ed889745297c35d
Jul 24 12:45:33.135782 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.135759    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 12:45:33.346205 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.346176    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:33.447124 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.446993    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:33.460701 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.460671    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"]
Jul 24 12:45:33.469718 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.469681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"
Jul 24 12:45:33.473399 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.473364    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-227c45ac-dockercfg-jgvc2\""
Jul 24 12:45:33.473566 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.473492    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-227c45ac\""
Jul 24 12:45:33.481797 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.481765    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"]
Jul 24 12:45:33.501627 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.501574    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"]
Jul 24 12:45:33.506592 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.506568    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"
Jul 24 12:45:33.518603 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.518573    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"]
Jul 24 12:45:33.539047 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.539000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4dr7\" (UniqueName: \"kubernetes.io/projected/03556062-a97e-458b-b0ca-0a949ca77e03-kube-api-access-k4dr7\") pod \"payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5\" (UID: \"03556062-a97e-458b-b0ca-0a949ca77e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"
Jul 24 12:45:33.539234 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.539210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03556062-a97e-458b-b0ca-0a949ca77e03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5\" (UID: \"03556062-a97e-458b-b0ca-0a949ca77e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"
Jul 24 12:45:33.581592 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.581558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9" event={"ID":"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5","Type":"ContainerStarted","Data":"5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2"}
Jul 24 12:45:33.581592 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.581611    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9" event={"ID":"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5","Type":"ContainerStarted","Data":"58e3001b9dcf8798c0e7c5e2f5841cb60259561cdf47fdb74ed889745297c35d"}
Jul 24 12:45:33.582270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.581658    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:33.603062 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.602973    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9" podStartSLOduration=1.2521727550000001 podStartE2EDuration="1.602950066s" totalImagesPullingTime="350.777311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:45:33.602254112 +0000 UTC m=+1607.130001161" watchObservedRunningTime="2026-07-24 12:45:33.602950066 +0000 UTC m=+1607.130697121"
Jul 24 12:45:33.640367 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.640320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4dr7\" (UniqueName: \"kubernetes.io/projected/03556062-a97e-458b-b0ca-0a949ca77e03-kube-api-access-k4dr7\") pod \"payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5\" (UID: \"03556062-a97e-458b-b0ca-0a949ca77e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"
Jul 24 12:45:33.640572 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.640435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6098513a-c2aa-4a20-a905-67b3d89b172c-plugins-config-volume\") pod \"payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6\" (UID: \"6098513a-c2aa-4a20-a905-67b3d89b172c\") " pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"
Jul 24 12:45:33.640572 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.640483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03556062-a97e-458b-b0ca-0a949ca77e03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5\" (UID: \"03556062-a97e-458b-b0ca-0a949ca77e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"
Jul 24 12:45:33.640572 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.640509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wktmk\" (UniqueName: \"kubernetes.io/projected/6098513a-c2aa-4a20-a905-67b3d89b172c-kube-api-access-wktmk\") pod \"payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6\" (UID: \"6098513a-c2aa-4a20-a905-67b3d89b172c\") " pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"
Jul 24 12:45:33.641261 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.641241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03556062-a97e-458b-b0ca-0a949ca77e03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5\" (UID: \"03556062-a97e-458b-b0ca-0a949ca77e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"
Jul 24 12:45:33.653376 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.653351    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4dr7\" (UniqueName: \"kubernetes.io/projected/03556062-a97e-458b-b0ca-0a949ca77e03-kube-api-access-k4dr7\") pod \"payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5\" (UID: \"03556062-a97e-458b-b0ca-0a949ca77e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"
Jul 24 12:45:33.741369 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.741266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6098513a-c2aa-4a20-a905-67b3d89b172c-plugins-config-volume\") pod \"payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6\" (UID: \"6098513a-c2aa-4a20-a905-67b3d89b172c\") " pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"
Jul 24 12:45:33.741369 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.741323    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wktmk\" (UniqueName: \"kubernetes.io/projected/6098513a-c2aa-4a20-a905-67b3d89b172c-kube-api-access-wktmk\") pod \"payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6\" (UID: \"6098513a-c2aa-4a20-a905-67b3d89b172c\") " pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"
Jul 24 12:45:33.742083 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.742061    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6098513a-c2aa-4a20-a905-67b3d89b172c-plugins-config-volume\") pod \"payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6\" (UID: \"6098513a-c2aa-4a20-a905-67b3d89b172c\") " pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"
Jul 24 12:45:33.750146 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.750118    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wktmk\" (UniqueName: \"kubernetes.io/projected/6098513a-c2aa-4a20-a905-67b3d89b172c-kube-api-access-wktmk\") pod \"payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6\" (UID: \"6098513a-c2aa-4a20-a905-67b3d89b172c\") " pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"
Jul 24 12:45:33.783977 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.783936    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"
Jul 24 12:45:33.832446 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.832413    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"
Jul 24 12:45:33.946902 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:33.946869    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"]
Jul 24 12:45:33.948902 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:45:33.948848    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03556062_a97e_458b_b0ca_0a949ca77e03.slice/crio-9dd1a49e862862f071a5d20f874e134d8dc8b9cedab28d2df5067588db293626 WatchSource:0}: Error finding container 9dd1a49e862862f071a5d20f874e134d8dc8b9cedab28d2df5067588db293626: Status 404 returned error can't find the container with id 9dd1a49e862862f071a5d20f874e134d8dc8b9cedab28d2df5067588db293626
Jul 24 12:45:34.003400 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:34.003237    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"]
Jul 24 12:45:34.005933 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:45:34.005906    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6098513a_c2aa_4a20_a905_67b3d89b172c.slice/crio-899c5e201ad17b03f1ffa6e822491c7aa9ecb41990d955a553ba75e3f52d4b62 WatchSource:0}: Error finding container 899c5e201ad17b03f1ffa6e822491c7aa9ecb41990d955a553ba75e3f52d4b62: Status 404 returned error can't find the container with id 899c5e201ad17b03f1ffa6e822491c7aa9ecb41990d955a553ba75e3f52d4b62
Jul 24 12:45:34.588659 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:34.588622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5" event={"ID":"03556062-a97e-458b-b0ca-0a949ca77e03","Type":"ContainerStarted","Data":"e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4"}
Jul 24 12:45:34.589183 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:34.588689    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"
Jul 24 12:45:34.589183 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:34.588717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5" event={"ID":"03556062-a97e-458b-b0ca-0a949ca77e03","Type":"ContainerStarted","Data":"9dd1a49e862862f071a5d20f874e134d8dc8b9cedab28d2df5067588db293626"}
Jul 24 12:45:34.590423 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:34.590394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6" event={"ID":"6098513a-c2aa-4a20-a905-67b3d89b172c","Type":"ContainerStarted","Data":"12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08"}
Jul 24 12:45:34.590563 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:34.590488    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"
Jul 24 12:45:34.590563 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:34.590527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6" event={"ID":"6098513a-c2aa-4a20-a905-67b3d89b172c","Type":"ContainerStarted","Data":"899c5e201ad17b03f1ffa6e822491c7aa9ecb41990d955a553ba75e3f52d4b62"}
Jul 24 12:45:34.608787 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:34.608722    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5" podStartSLOduration=1.608701459 podStartE2EDuration="1.608701459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:45:34.606045471 +0000 UTC m=+1608.133792527" watchObservedRunningTime="2026-07-24 12:45:34.608701459 +0000 UTC m=+1608.136448515"
Jul 24 12:45:34.629264 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:34.629207    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6" podStartSLOduration=1.629189167 podStartE2EDuration="1.629189167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:45:34.627476936 +0000 UTC m=+1608.155223996" watchObservedRunningTime="2026-07-24 12:45:34.629189167 +0000 UTC m=+1608.156936223"
Jul 24 12:45:39.596222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:39.596143    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:45.598163 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:45.598129    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"
Jul 24 12:45:45.598585 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:45.598207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"
Jul 24 12:45:53.292637 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.292596    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"]
Jul 24 12:45:53.293263 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.292871    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9" podUID="35a8731a-d9bc-41cd-9a00-98cc50f0f6a5" containerName="maas-api" containerID="cri-o://5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2" gracePeriod=30
Jul 24 12:45:53.325955 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.325905    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:53.342412 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.342374    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"]
Jul 24 12:45:53.342618 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.342592    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6" podUID="6098513a-c2aa-4a20-a905-67b3d89b172c" containerName="payload-processing" containerID="cri-o://12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08" gracePeriod=30
Jul 24 12:45:53.358000 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.357966    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"]
Jul 24 12:45:53.358392 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.358341    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5" podUID="03556062-a97e-458b-b0ca-0a949ca77e03" containerName="payload-pre-processing" containerID="cri-o://e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4" gracePeriod=30
Jul 24 12:45:53.632001 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.631971    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"
Jul 24 12:45:53.672909 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.672877    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:53.682645 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.682615    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"
Jul 24 12:45:53.690171 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.690136    2566 generic.go:356] "Generic (PLEG): container finished" podID="35a8731a-d9bc-41cd-9a00-98cc50f0f6a5" containerID="5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2" exitCode=0
Jul 24 12:45:53.690347 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.690234    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"
Jul 24 12:45:53.690347 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.690255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9" event={"ID":"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5","Type":"ContainerDied","Data":"5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2"}
Jul 24 12:45:53.690473 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.690350    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9" event={"ID":"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5","Type":"ContainerDied","Data":"58e3001b9dcf8798c0e7c5e2f5841cb60259561cdf47fdb74ed889745297c35d"}
Jul 24 12:45:53.690473 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.690375    2566 scope.go:117] "RemoveContainer" containerID="5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2"
Jul 24 12:45:53.691761 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.691723    2566 generic.go:356] "Generic (PLEG): container finished" podID="03556062-a97e-458b-b0ca-0a949ca77e03" containerID="e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4" exitCode=0
Jul 24 12:45:53.691912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.691902    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"
Jul 24 12:45:53.692512 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.692110    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5" event={"ID":"03556062-a97e-458b-b0ca-0a949ca77e03","Type":"ContainerDied","Data":"e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4"}
Jul 24 12:45:53.692512 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.692180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5" event={"ID":"03556062-a97e-458b-b0ca-0a949ca77e03","Type":"ContainerDied","Data":"9dd1a49e862862f071a5d20f874e134d8dc8b9cedab28d2df5067588db293626"}
Jul 24 12:45:53.693559 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.693533    2566 generic.go:356] "Generic (PLEG): container finished" podID="6098513a-c2aa-4a20-a905-67b3d89b172c" containerID="12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08" exitCode=0
Jul 24 12:45:53.693680 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.693573    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"
Jul 24 12:45:53.693680 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.693608    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6" event={"ID":"6098513a-c2aa-4a20-a905-67b3d89b172c","Type":"ContainerDied","Data":"12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08"}
Jul 24 12:45:53.693680 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.693648    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6" event={"ID":"6098513a-c2aa-4a20-a905-67b3d89b172c","Type":"ContainerDied","Data":"899c5e201ad17b03f1ffa6e822491c7aa9ecb41990d955a553ba75e3f52d4b62"}
Jul 24 12:45:53.701124 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.701104    2566 scope.go:117] "RemoveContainer" containerID="5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2"
Jul 24 12:45:53.701434 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:45:53.701411    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2\": container with ID starting with 5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2 not found: ID does not exist" containerID="5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2"
Jul 24 12:45:53.701508 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.701442    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2"} err="failed to get container status \"5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2\": rpc error: code = NotFound desc = could not find container \"5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2\": container with ID starting with 5c6a2699a17cfb552a2d061f21f5224170d0c0404655eb661582bff913138bb2 not found: ID does not exist"
Jul 24 12:45:53.701508 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.701460    2566 scope.go:117] "RemoveContainer" containerID="e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4"
Jul 24 12:45:53.712355 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.712222    2566 scope.go:117] "RemoveContainer" containerID="e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4"
Jul 24 12:45:53.713297 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:45:53.713267    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4\": container with ID starting with e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4 not found: ID does not exist" containerID="e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4"
Jul 24 12:45:53.713417 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.713309    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4"} err="failed to get container status \"e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4\": rpc error: code = NotFound desc = could not find container \"e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4\": container with ID starting with e66070c4d5c1f382127dde9f5b8f182bd71e5f272af7adae27be4cfce36bf7c4 not found: ID does not exist"
Jul 24 12:45:53.713417 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.713334    2566 scope.go:117] "RemoveContainer" containerID="12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08"
Jul 24 12:45:53.725106 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.725084    2566 scope.go:117] "RemoveContainer" containerID="12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08"
Jul 24 12:45:53.725429 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:45:53.725407    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08\": container with ID starting with 12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08 not found: ID does not exist" containerID="12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08"
Jul 24 12:45:53.725506 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.725437    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08"} err="failed to get container status \"12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08\": rpc error: code = NotFound desc = could not find container \"12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08\": container with ID starting with 12ceb0695a0b276347ebbc18375dad7f73f9730817b674109e7e764982794d08 not found: ID does not exist"
Jul 24 12:45:53.739009 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.738984    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-openshift-service-ca\") pod \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\" (UID: \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\") "
Jul 24 12:45:53.739169 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.739043    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wktmk\" (UniqueName: \"kubernetes.io/projected/6098513a-c2aa-4a20-a905-67b3d89b172c-kube-api-access-wktmk\") pod \"6098513a-c2aa-4a20-a905-67b3d89b172c\" (UID: \"6098513a-c2aa-4a20-a905-67b3d89b172c\") "
Jul 24 12:45:53.739169 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.739063    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkgqm\" (UniqueName: \"kubernetes.io/projected/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-kube-api-access-hkgqm\") pod \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\" (UID: \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\") "
Jul 24 12:45:53.739370 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.739245    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-maas-api-tls\") pod \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\" (UID: \"35a8731a-d9bc-41cd-9a00-98cc50f0f6a5\") "
Jul 24 12:45:53.739370 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.739341    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03556062-a97e-458b-b0ca-0a949ca77e03-plugins-config-volume\") pod \"03556062-a97e-458b-b0ca-0a949ca77e03\" (UID: \"03556062-a97e-458b-b0ca-0a949ca77e03\") "
Jul 24 12:45:53.739481 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.739397    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4dr7\" (UniqueName: \"kubernetes.io/projected/03556062-a97e-458b-b0ca-0a949ca77e03-kube-api-access-k4dr7\") pod \"03556062-a97e-458b-b0ca-0a949ca77e03\" (UID: \"03556062-a97e-458b-b0ca-0a949ca77e03\") "
Jul 24 12:45:53.739481 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.739451    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6098513a-c2aa-4a20-a905-67b3d89b172c-plugins-config-volume\") pod \"6098513a-c2aa-4a20-a905-67b3d89b172c\" (UID: \"6098513a-c2aa-4a20-a905-67b3d89b172c\") "
Jul 24 12:45:53.739887 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.739863    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "35a8731a-d9bc-41cd-9a00-98cc50f0f6a5" (UID: "35a8731a-d9bc-41cd-9a00-98cc50f0f6a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:45:53.740058 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.740034    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03556062-a97e-458b-b0ca-0a949ca77e03-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03556062-a97e-458b-b0ca-0a949ca77e03" (UID: "03556062-a97e-458b-b0ca-0a949ca77e03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:45:53.740153 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.740131    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6098513a-c2aa-4a20-a905-67b3d89b172c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6098513a-c2aa-4a20-a905-67b3d89b172c" (UID: "6098513a-c2aa-4a20-a905-67b3d89b172c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:45:53.741884 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.741851    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03556062-a97e-458b-b0ca-0a949ca77e03-kube-api-access-k4dr7" (OuterVolumeSpecName: "kube-api-access-k4dr7") pod "03556062-a97e-458b-b0ca-0a949ca77e03" (UID: "03556062-a97e-458b-b0ca-0a949ca77e03"). InnerVolumeSpecName "kube-api-access-k4dr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:45:53.741983 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.741936    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6098513a-c2aa-4a20-a905-67b3d89b172c-kube-api-access-wktmk" (OuterVolumeSpecName: "kube-api-access-wktmk") pod "6098513a-c2aa-4a20-a905-67b3d89b172c" (UID: "6098513a-c2aa-4a20-a905-67b3d89b172c"). InnerVolumeSpecName "kube-api-access-wktmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:45:53.742572 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.742543    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "35a8731a-d9bc-41cd-9a00-98cc50f0f6a5" (UID: "35a8731a-d9bc-41cd-9a00-98cc50f0f6a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:45:53.742653 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.742569    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-kube-api-access-hkgqm" (OuterVolumeSpecName: "kube-api-access-hkgqm") pod "35a8731a-d9bc-41cd-9a00-98cc50f0f6a5" (UID: "35a8731a-d9bc-41cd-9a00-98cc50f0f6a5"). InnerVolumeSpecName "kube-api-access-hkgqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:45:53.840672 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.840566    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03556062-a97e-458b-b0ca-0a949ca77e03-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:45:53.840672 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.840610    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4dr7\" (UniqueName: \"kubernetes.io/projected/03556062-a97e-458b-b0ca-0a949ca77e03-kube-api-access-k4dr7\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:45:53.840672 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.840620    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6098513a-c2aa-4a20-a905-67b3d89b172c-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:45:53.840672 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.840632    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:45:53.840672 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.840642    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wktmk\" (UniqueName: \"kubernetes.io/projected/6098513a-c2aa-4a20-a905-67b3d89b172c-kube-api-access-wktmk\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:45:53.840672 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.840650    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkgqm\" (UniqueName: \"kubernetes.io/projected/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-kube-api-access-hkgqm\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:45:53.840672 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:53.840659    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:45:54.020048 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:54.019980    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"]
Jul 24 12:45:54.024633 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:54.024598    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-227c45ac-545d7c9d67-wbgk9"]
Jul 24 12:45:54.036091 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:54.036055    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"]
Jul 24 12:45:54.039788 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:54.039745    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-227c45ac-659b94b9d-jb4q6"]
Jul 24 12:45:54.051732 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:54.051698    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"]
Jul 24 12:45:54.057627 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:54.057596    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-227c45ac-7cc8649776-9rfz5"]
Jul 24 12:45:54.975388 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:54.975354    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03556062-a97e-458b-b0ca-0a949ca77e03" path="/var/lib/kubelet/pods/03556062-a97e-458b-b0ca-0a949ca77e03/volumes"
Jul 24 12:45:54.975780 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:54.975742    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a8731a-d9bc-41cd-9a00-98cc50f0f6a5" path="/var/lib/kubelet/pods/35a8731a-d9bc-41cd-9a00-98cc50f0f6a5/volumes"
Jul 24 12:45:54.976089 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:54.976075    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6098513a-c2aa-4a20-a905-67b3d89b172c" path="/var/lib/kubelet/pods/6098513a-c2aa-4a20-a905-67b3d89b172c/volumes"
Jul 24 12:45:54.998081 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:54.998054    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:57.900208 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:57.900171    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:58.072289 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:58.072257    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:58.316171 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:58.316129    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:58.599278 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:58.599182    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:58.699219 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:58.699185    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:58.802115 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:58.802079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:59.134338 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:59.134294    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:45:59.906388 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:45:59.906350    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:01.836910 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.836866    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:01.922831 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.922797    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"]
Jul 24 12:46:01.923379 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.923328    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a8731a-d9bc-41cd-9a00-98cc50f0f6a5" containerName="maas-api"
Jul 24 12:46:01.923379 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.923354    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a8731a-d9bc-41cd-9a00-98cc50f0f6a5" containerName="maas-api"
Jul 24 12:46:01.923379 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.923380    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03556062-a97e-458b-b0ca-0a949ca77e03" containerName="payload-pre-processing"
Jul 24 12:46:01.923726 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.923388    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03556062-a97e-458b-b0ca-0a949ca77e03" containerName="payload-pre-processing"
Jul 24 12:46:01.923726 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.923418    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6098513a-c2aa-4a20-a905-67b3d89b172c" containerName="payload-processing"
Jul 24 12:46:01.923726 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.923426    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6098513a-c2aa-4a20-a905-67b3d89b172c" containerName="payload-processing"
Jul 24 12:46:01.923726 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.923525    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a8731a-d9bc-41cd-9a00-98cc50f0f6a5" containerName="maas-api"
Jul 24 12:46:01.923726 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.923537    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6098513a-c2aa-4a20-a905-67b3d89b172c" containerName="payload-processing"
Jul 24 12:46:01.923726 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.923548    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03556062-a97e-458b-b0ca-0a949ca77e03" containerName="payload-pre-processing"
Jul 24 12:46:01.928218 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.928195    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:01.931184 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.931166    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b0a74db1-serving-cert\""
Jul 24 12:46:01.938574 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.938544    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"]
Jul 24 12:46:01.959523 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:01.959473    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:02.020696 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.020650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b75a11-a1b6-4773-81fd-103a860defa3-openshift-service-ca\") pod \"maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:02.020905 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.020748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52b75a11-a1b6-4773-81fd-103a860defa3-maas-api-tls\") pod \"maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:02.020905 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.020769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749ff\" (UniqueName: \"kubernetes.io/projected/52b75a11-a1b6-4773-81fd-103a860defa3-kube-api-access-749ff\") pod \"maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:02.121506 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.121404    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b75a11-a1b6-4773-81fd-103a860defa3-openshift-service-ca\") pod \"maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:02.121506 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.121492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52b75a11-a1b6-4773-81fd-103a860defa3-maas-api-tls\") pod \"maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:02.121729 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.121515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749ff\" (UniqueName: \"kubernetes.io/projected/52b75a11-a1b6-4773-81fd-103a860defa3-kube-api-access-749ff\") pod \"maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:02.121729 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:46:02.121642    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-serving-cert: secret "maas-api-e2e-migrate-b0a74db1-serving-cert" not found
Jul 24 12:46:02.121729 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:46:02.121714    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b75a11-a1b6-4773-81fd-103a860defa3-maas-api-tls podName:52b75a11-a1b6-4773-81fd-103a860defa3 nodeName:}" failed. No retries permitted until 2026-07-24 12:46:02.621694062 +0000 UTC m=+1636.149441109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52b75a11-a1b6-4773-81fd-103a860defa3-maas-api-tls") pod "maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv" (UID: "52b75a11-a1b6-4773-81fd-103a860defa3") : secret "maas-api-e2e-migrate-b0a74db1-serving-cert" not found
Jul 24 12:46:02.122119 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.122100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b75a11-a1b6-4773-81fd-103a860defa3-openshift-service-ca\") pod \"maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:02.132439 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.132407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-749ff\" (UniqueName: \"kubernetes.io/projected/52b75a11-a1b6-4773-81fd-103a860defa3-kube-api-access-749ff\") pod \"maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:02.576391 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.576354    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:02.626364 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.626323    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52b75a11-a1b6-4773-81fd-103a860defa3-maas-api-tls\") pod \"maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:02.629337 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.629296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52b75a11-a1b6-4773-81fd-103a860defa3-maas-api-tls\") pod \"maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:02.674411 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.674377    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"]
Jul 24 12:46:02.677911 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.677895    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"
Jul 24 12:46:02.680888 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.680864    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b0a74db1-dockercfg-kgn5k\""
Jul 24 12:46:02.681074 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.681042    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b0a74db1\""
Jul 24 12:46:02.688450 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.688429    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"]
Jul 24 12:46:02.692472 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.692447    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"]
Jul 24 12:46:02.696628 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.696611    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"
Jul 24 12:46:02.710037 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.709986    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"]
Jul 24 12:46:02.741987 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.741949    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:02.801726 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.801689    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:02.827942 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.827837    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbf69d3-7d86-4069-8bc5-2cf260b72e58-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw\" (UID: \"bbbf69d3-7d86-4069-8bc5-2cf260b72e58\") " pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"
Jul 24 12:46:02.827942 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.827896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zscpl\" (UniqueName: \"kubernetes.io/projected/5ae06385-fc39-46c7-b5ab-801148c96a71-kube-api-access-zscpl\") pod \"payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw\" (UID: \"5ae06385-fc39-46c7-b5ab-801148c96a71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"
Jul 24 12:46:02.828186 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.828053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae06385-fc39-46c7-b5ab-801148c96a71-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw\" (UID: \"5ae06385-fc39-46c7-b5ab-801148c96a71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"
Jul 24 12:46:02.828186 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.828092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkf6x\" (UniqueName: \"kubernetes.io/projected/bbbf69d3-7d86-4069-8bc5-2cf260b72e58-kube-api-access-jkf6x\") pod \"payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw\" (UID: \"bbbf69d3-7d86-4069-8bc5-2cf260b72e58\") " pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"
Jul 24 12:46:02.840945 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.840917    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:02.929290 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.929246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae06385-fc39-46c7-b5ab-801148c96a71-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw\" (UID: \"5ae06385-fc39-46c7-b5ab-801148c96a71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"
Jul 24 12:46:02.929502 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.929303    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkf6x\" (UniqueName: \"kubernetes.io/projected/bbbf69d3-7d86-4069-8bc5-2cf260b72e58-kube-api-access-jkf6x\") pod \"payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw\" (UID: \"bbbf69d3-7d86-4069-8bc5-2cf260b72e58\") " pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"
Jul 24 12:46:02.929502 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.929384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbf69d3-7d86-4069-8bc5-2cf260b72e58-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw\" (UID: \"bbbf69d3-7d86-4069-8bc5-2cf260b72e58\") " pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"
Jul 24 12:46:02.929502 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.929423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zscpl\" (UniqueName: \"kubernetes.io/projected/5ae06385-fc39-46c7-b5ab-801148c96a71-kube-api-access-zscpl\") pod \"payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw\" (UID: \"5ae06385-fc39-46c7-b5ab-801148c96a71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"
Jul 24 12:46:02.929947 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.929921    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae06385-fc39-46c7-b5ab-801148c96a71-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw\" (UID: \"5ae06385-fc39-46c7-b5ab-801148c96a71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"
Jul 24 12:46:02.930534 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.930514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbf69d3-7d86-4069-8bc5-2cf260b72e58-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw\" (UID: \"bbbf69d3-7d86-4069-8bc5-2cf260b72e58\") " pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"
Jul 24 12:46:02.939424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.939359    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zscpl\" (UniqueName: \"kubernetes.io/projected/5ae06385-fc39-46c7-b5ab-801148c96a71-kube-api-access-zscpl\") pod \"payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw\" (UID: \"5ae06385-fc39-46c7-b5ab-801148c96a71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"
Jul 24 12:46:02.939820 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.939797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkf6x\" (UniqueName: \"kubernetes.io/projected/bbbf69d3-7d86-4069-8bc5-2cf260b72e58-kube-api-access-jkf6x\") pod \"payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw\" (UID: \"bbbf69d3-7d86-4069-8bc5-2cf260b72e58\") " pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"
Jul 24 12:46:02.989721 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:02.989684    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"
Jul 24 12:46:03.002591 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.002557    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"]
Jul 24 12:46:03.004816 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:46:03.004719    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b75a11_a1b6_4773_81fd_103a860defa3.slice/crio-8ce0214749f3053777837478281fbda387ad75cefde1a4180d3bc7a0d2513fbb WatchSource:0}: Error finding container 8ce0214749f3053777837478281fbda387ad75cefde1a4180d3bc7a0d2513fbb: Status 404 returned error can't find the container with id 8ce0214749f3053777837478281fbda387ad75cefde1a4180d3bc7a0d2513fbb
Jul 24 12:46:03.007872 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.007843    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"
Jul 24 12:46:03.157784 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.157208    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"]
Jul 24 12:46:03.189223 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.189199    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"]
Jul 24 12:46:03.191828 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:46:03.191788    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbf69d3_7d86_4069_8bc5_2cf260b72e58.slice/crio-cb80ede8d80a6c829014143ac974f9c71e59946ef30ec190c9af5535b7490958 WatchSource:0}: Error finding container cb80ede8d80a6c829014143ac974f9c71e59946ef30ec190c9af5535b7490958: Status 404 returned error can't find the container with id cb80ede8d80a6c829014143ac974f9c71e59946ef30ec190c9af5535b7490958
Jul 24 12:46:03.752544 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.752509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw" event={"ID":"bbbf69d3-7d86-4069-8bc5-2cf260b72e58","Type":"ContainerStarted","Data":"e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d"}
Jul 24 12:46:03.752732 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.752602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw" event={"ID":"bbbf69d3-7d86-4069-8bc5-2cf260b72e58","Type":"ContainerStarted","Data":"cb80ede8d80a6c829014143ac974f9c71e59946ef30ec190c9af5535b7490958"}
Jul 24 12:46:03.752732 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.752633    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"
Jul 24 12:46:03.754253 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.754224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv" event={"ID":"52b75a11-a1b6-4773-81fd-103a860defa3","Type":"ContainerStarted","Data":"90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a"}
Jul 24 12:46:03.754384 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.754277    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv" event={"ID":"52b75a11-a1b6-4773-81fd-103a860defa3","Type":"ContainerStarted","Data":"8ce0214749f3053777837478281fbda387ad75cefde1a4180d3bc7a0d2513fbb"}
Jul 24 12:46:03.754384 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.754308    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:03.755806 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.755779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw" event={"ID":"5ae06385-fc39-46c7-b5ab-801148c96a71","Type":"ContainerStarted","Data":"9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b"}
Jul 24 12:46:03.755921 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.755830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw" event={"ID":"5ae06385-fc39-46c7-b5ab-801148c96a71","Type":"ContainerStarted","Data":"c1e29c51c7e011be08d035d75d74b9934b0dd97fe85a17af618299487da88bcc"}
Jul 24 12:46:03.755921 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.755904    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"
Jul 24 12:46:03.772100 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.772008    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw" podStartSLOduration=1.7719862210000001 podStartE2EDuration="1.771986221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:46:03.770722578 +0000 UTC m=+1637.298469633" watchObservedRunningTime="2026-07-24 12:46:03.771986221 +0000 UTC m=+1637.299733279"
Jul 24 12:46:03.789846 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.789780    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv" podStartSLOduration=2.407317994 podStartE2EDuration="2.789757974s" totalImagesPullingTime="382.43998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:46:03.787392722 +0000 UTC m=+1637.315139776" watchObservedRunningTime="2026-07-24 12:46:03.789757974 +0000 UTC m=+1637.317505029"
Jul 24 12:46:03.807702 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:03.807643    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw" podStartSLOduration=1.8076252400000001 podStartE2EDuration="1.80762524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:46:03.805462963 +0000 UTC m=+1637.333210017" watchObservedRunningTime="2026-07-24 12:46:03.80762524 +0000 UTC m=+1637.335372293"
Jul 24 12:46:09.768131 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:09.768099    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:14.763270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:14.763241    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"
Jul 24 12:46:14.763856 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:14.763329    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"
Jul 24 12:46:37.433810 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.433770    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"]
Jul 24 12:46:37.434373 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.434061    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv" podUID="52b75a11-a1b6-4773-81fd-103a860defa3" containerName="maas-api" containerID="cri-o://90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a" gracePeriod=30
Jul 24 12:46:37.464182 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.464150    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:37.494181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.494145    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"]
Jul 24 12:46:37.494474 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.494418    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw" podUID="bbbf69d3-7d86-4069-8bc5-2cf260b72e58" containerName="payload-processing" containerID="cri-o://e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d" gracePeriod=30
Jul 24 12:46:37.502971 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.502939    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"]
Jul 24 12:46:37.503227 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.503200    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw" podUID="5ae06385-fc39-46c7-b5ab-801148c96a71" containerName="payload-pre-processing" containerID="cri-o://9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b" gracePeriod=30
Jul 24 12:46:37.537184 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.537151    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:37.621203 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.621155    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:37.790088 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.790061    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:37.794377 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.794353    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"
Jul 24 12:46:37.811358 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.811331    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"
Jul 24 12:46:37.861472 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.861433    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52b75a11-a1b6-4773-81fd-103a860defa3-maas-api-tls\") pod \"52b75a11-a1b6-4773-81fd-103a860defa3\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") "
Jul 24 12:46:37.861673 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.861500    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b75a11-a1b6-4773-81fd-103a860defa3-openshift-service-ca\") pod \"52b75a11-a1b6-4773-81fd-103a860defa3\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") "
Jul 24 12:46:37.861673 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.861540    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkf6x\" (UniqueName: \"kubernetes.io/projected/bbbf69d3-7d86-4069-8bc5-2cf260b72e58-kube-api-access-jkf6x\") pod \"bbbf69d3-7d86-4069-8bc5-2cf260b72e58\" (UID: \"bbbf69d3-7d86-4069-8bc5-2cf260b72e58\") "
Jul 24 12:46:37.861673 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.861566    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-749ff\" (UniqueName: \"kubernetes.io/projected/52b75a11-a1b6-4773-81fd-103a860defa3-kube-api-access-749ff\") pod \"52b75a11-a1b6-4773-81fd-103a860defa3\" (UID: \"52b75a11-a1b6-4773-81fd-103a860defa3\") "
Jul 24 12:46:37.861673 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.861612    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbf69d3-7d86-4069-8bc5-2cf260b72e58-plugins-config-volume\") pod \"bbbf69d3-7d86-4069-8bc5-2cf260b72e58\" (UID: \"bbbf69d3-7d86-4069-8bc5-2cf260b72e58\") "
Jul 24 12:46:37.861996 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.861903    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b75a11-a1b6-4773-81fd-103a860defa3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52b75a11-a1b6-4773-81fd-103a860defa3" (UID: "52b75a11-a1b6-4773-81fd-103a860defa3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:46:37.862274 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.862247    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbf69d3-7d86-4069-8bc5-2cf260b72e58-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbbf69d3-7d86-4069-8bc5-2cf260b72e58" (UID: "bbbf69d3-7d86-4069-8bc5-2cf260b72e58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:46:37.864273 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.864243    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b75a11-a1b6-4773-81fd-103a860defa3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52b75a11-a1b6-4773-81fd-103a860defa3" (UID: "52b75a11-a1b6-4773-81fd-103a860defa3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:46:37.864403 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.864344    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbf69d3-7d86-4069-8bc5-2cf260b72e58-kube-api-access-jkf6x" (OuterVolumeSpecName: "kube-api-access-jkf6x") pod "bbbf69d3-7d86-4069-8bc5-2cf260b72e58" (UID: "bbbf69d3-7d86-4069-8bc5-2cf260b72e58"). InnerVolumeSpecName "kube-api-access-jkf6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:46:37.864548 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.864515    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b75a11-a1b6-4773-81fd-103a860defa3-kube-api-access-749ff" (OuterVolumeSpecName: "kube-api-access-749ff") pod "52b75a11-a1b6-4773-81fd-103a860defa3" (UID: "52b75a11-a1b6-4773-81fd-103a860defa3"). InnerVolumeSpecName "kube-api-access-749ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:46:37.932697 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.932652    2566 generic.go:356] "Generic (PLEG): container finished" podID="5ae06385-fc39-46c7-b5ab-801148c96a71" containerID="9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b" exitCode=0
Jul 24 12:46:37.932697 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.932702    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"
Jul 24 12:46:37.932928 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.932741    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw" event={"ID":"5ae06385-fc39-46c7-b5ab-801148c96a71","Type":"ContainerDied","Data":"9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b"}
Jul 24 12:46:37.932928 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.932803    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw" event={"ID":"5ae06385-fc39-46c7-b5ab-801148c96a71","Type":"ContainerDied","Data":"c1e29c51c7e011be08d035d75d74b9934b0dd97fe85a17af618299487da88bcc"}
Jul 24 12:46:37.932928 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.932838    2566 scope.go:117] "RemoveContainer" containerID="9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b"
Jul 24 12:46:37.934164 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.934136    2566 generic.go:356] "Generic (PLEG): container finished" podID="bbbf69d3-7d86-4069-8bc5-2cf260b72e58" containerID="e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d" exitCode=0
Jul 24 12:46:37.934308 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.934193    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"
Jul 24 12:46:37.934308 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.934240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw" event={"ID":"bbbf69d3-7d86-4069-8bc5-2cf260b72e58","Type":"ContainerDied","Data":"e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d"}
Jul 24 12:46:37.934308 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.934288    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw" event={"ID":"bbbf69d3-7d86-4069-8bc5-2cf260b72e58","Type":"ContainerDied","Data":"cb80ede8d80a6c829014143ac974f9c71e59946ef30ec190c9af5535b7490958"}
Jul 24 12:46:37.935664 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.935641    2566 generic.go:356] "Generic (PLEG): container finished" podID="52b75a11-a1b6-4773-81fd-103a860defa3" containerID="90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a" exitCode=0
Jul 24 12:46:37.935763 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.935684    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"
Jul 24 12:46:37.935763 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.935685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv" event={"ID":"52b75a11-a1b6-4773-81fd-103a860defa3","Type":"ContainerDied","Data":"90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a"}
Jul 24 12:46:37.935853 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.935806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv" event={"ID":"52b75a11-a1b6-4773-81fd-103a860defa3","Type":"ContainerDied","Data":"8ce0214749f3053777837478281fbda387ad75cefde1a4180d3bc7a0d2513fbb"}
Jul 24 12:46:37.943999 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.943975    2566 scope.go:117] "RemoveContainer" containerID="9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b"
Jul 24 12:46:37.944301 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:46:37.944282    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b\": container with ID starting with 9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b not found: ID does not exist" containerID="9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b"
Jul 24 12:46:37.944368 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.944310    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b"} err="failed to get container status \"9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b\": rpc error: code = NotFound desc = could not find container \"9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b\": container with ID starting with 9f4a9d2266a65d56fa07fc46d1dd03a6eb74eede5fe57088550ec6fbdc6bed4b not found: ID does not exist"
Jul 24 12:46:37.944368 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.944326    2566 scope.go:117] "RemoveContainer" containerID="e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d"
Jul 24 12:46:37.952838 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.952823    2566 scope.go:117] "RemoveContainer" containerID="e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d"
Jul 24 12:46:37.953113 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:46:37.953092    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d\": container with ID starting with e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d not found: ID does not exist" containerID="e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d"
Jul 24 12:46:37.953184 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.953118    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d"} err="failed to get container status \"e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d\": rpc error: code = NotFound desc = could not find container \"e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d\": container with ID starting with e2c594fddf3fb8c131b0e3a4fd248abc9c1a628408a9ba3b7522984a944b765d not found: ID does not exist"
Jul 24 12:46:37.953184 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.953134    2566 scope.go:117] "RemoveContainer" containerID="90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a"
Jul 24 12:46:37.959262 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.959234    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"]
Jul 24 12:46:37.962233 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.962214    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zscpl\" (UniqueName: \"kubernetes.io/projected/5ae06385-fc39-46c7-b5ab-801148c96a71-kube-api-access-zscpl\") pod \"5ae06385-fc39-46c7-b5ab-801148c96a71\" (UID: \"5ae06385-fc39-46c7-b5ab-801148c96a71\") "
Jul 24 12:46:37.962308 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.962267    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae06385-fc39-46c7-b5ab-801148c96a71-plugins-config-volume\") pod \"5ae06385-fc39-46c7-b5ab-801148c96a71\" (UID: \"5ae06385-fc39-46c7-b5ab-801148c96a71\") "
Jul 24 12:46:37.962308 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.962221    2566 scope.go:117] "RemoveContainer" containerID="90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a"
Jul 24 12:46:37.962615 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.962589    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbf69d3-7d86-4069-8bc5-2cf260b72e58-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:46:37.962748 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.962623    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52b75a11-a1b6-4773-81fd-103a860defa3-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:46:37.962748 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.962637    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b75a11-a1b6-4773-81fd-103a860defa3-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:46:37.962748 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.962650    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkf6x\" (UniqueName: \"kubernetes.io/projected/bbbf69d3-7d86-4069-8bc5-2cf260b72e58-kube-api-access-jkf6x\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:46:37.962748 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.962664    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-749ff\" (UniqueName: \"kubernetes.io/projected/52b75a11-a1b6-4773-81fd-103a860defa3-kube-api-access-749ff\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:46:37.962748 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.962731    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae06385-fc39-46c7-b5ab-801148c96a71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ae06385-fc39-46c7-b5ab-801148c96a71" (UID: "5ae06385-fc39-46c7-b5ab-801148c96a71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:46:37.963145 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:46:37.963114    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a\": container with ID starting with 90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a not found: ID does not exist" containerID="90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a"
Jul 24 12:46:37.963248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.963155    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a"} err="failed to get container status \"90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a\": rpc error: code = NotFound desc = could not find container \"90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a\": container with ID starting with 90cfcdadc1535377214cb879c6cbe774d2d4e1d207ab81699247b5e716f8582a not found: ID does not exist"
Jul 24 12:46:37.964507 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.964488    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae06385-fc39-46c7-b5ab-801148c96a71-kube-api-access-zscpl" (OuterVolumeSpecName: "kube-api-access-zscpl") pod "5ae06385-fc39-46c7-b5ab-801148c96a71" (UID: "5ae06385-fc39-46c7-b5ab-801148c96a71"). InnerVolumeSpecName "kube-api-access-zscpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:46:37.965126 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.965104    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b0a74db1-5d597f9db5-5vzhw"]
Jul 24 12:46:37.976236 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.976215    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"]
Jul 24 12:46:37.982142 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:37.982119    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b0a74db1-7f5dff48db-mxltv"]
Jul 24 12:46:38.063763 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:38.063724    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zscpl\" (UniqueName: \"kubernetes.io/projected/5ae06385-fc39-46c7-b5ab-801148c96a71-kube-api-access-zscpl\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:46:38.063763 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:38.063757    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae06385-fc39-46c7-b5ab-801148c96a71-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:46:38.258204 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:38.258169    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"]
Jul 24 12:46:38.266366 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:38.266337    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b0a74db1-97c76db68-dkggw"]
Jul 24 12:46:38.975588 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:38.975553    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52b75a11-a1b6-4773-81fd-103a860defa3" path="/var/lib/kubelet/pods/52b75a11-a1b6-4773-81fd-103a860defa3/volumes"
Jul 24 12:46:38.975986 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:38.975910    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ae06385-fc39-46c7-b5ab-801148c96a71" path="/var/lib/kubelet/pods/5ae06385-fc39-46c7-b5ab-801148c96a71/volumes"
Jul 24 12:46:38.976269 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:38.976254    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbbf69d3-7d86-4069-8bc5-2cf260b72e58" path="/var/lib/kubelet/pods/bbbf69d3-7d86-4069-8bc5-2cf260b72e58/volumes"
Jul 24 12:46:42.045716 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:42.045673    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:42.228472 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:42.228431    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:42.444862 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:42.444763    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:42.702634 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:42.702547    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:43.088483 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:43.088447    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:43.387933 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:43.387833    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:45.053913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.053848    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:45.191573 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.191537    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:45.255553 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.255149    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"]
Jul 24 12:46:45.255800 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.255776    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b75a11-a1b6-4773-81fd-103a860defa3" containerName="maas-api"
Jul 24 12:46:45.255913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.255804    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b75a11-a1b6-4773-81fd-103a860defa3" containerName="maas-api"
Jul 24 12:46:45.255913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.255835    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae06385-fc39-46c7-b5ab-801148c96a71" containerName="payload-pre-processing"
Jul 24 12:46:45.255913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.255843    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae06385-fc39-46c7-b5ab-801148c96a71" containerName="payload-pre-processing"
Jul 24 12:46:45.255913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.255856    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbbf69d3-7d86-4069-8bc5-2cf260b72e58" containerName="payload-processing"
Jul 24 12:46:45.255913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.255864    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbbf69d3-7d86-4069-8bc5-2cf260b72e58" containerName="payload-processing"
Jul 24 12:46:45.256232 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.255966    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="52b75a11-a1b6-4773-81fd-103a860defa3" containerName="maas-api"
Jul 24 12:46:45.256232 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.255979    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae06385-fc39-46c7-b5ab-801148c96a71" containerName="payload-pre-processing"
Jul 24 12:46:45.256232 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.255988    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbbf69d3-7d86-4069-8bc5-2cf260b72e58" containerName="payload-processing"
Jul 24 12:46:45.259607 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.259581    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:45.262691 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.262665    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-52688321-serving-cert\""
Jul 24 12:46:45.300227 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.300175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"]
Jul 24 12:46:45.333482 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.333391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0523d973-2496-491c-b8d0-bc0558a31020-openshift-service-ca\") pod \"maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:45.333636 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.333491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0523d973-2496-491c-b8d0-bc0558a31020-maas-api-tls\") pod \"maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:45.333636 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.333578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s27m8\" (UniqueName: \"kubernetes.io/projected/0523d973-2496-491c-b8d0-bc0558a31020-kube-api-access-s27m8\") pod \"maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:45.376101 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.376061    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:45.435090 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.435009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0523d973-2496-491c-b8d0-bc0558a31020-maas-api-tls\") pod \"maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:45.435318 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.435153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s27m8\" (UniqueName: \"kubernetes.io/projected/0523d973-2496-491c-b8d0-bc0558a31020-kube-api-access-s27m8\") pod \"maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:45.435318 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:46:45.435164    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-52688321-serving-cert: secret "maas-api-e2e-ait-52688321-serving-cert" not found
Jul 24 12:46:45.435318 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.435190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0523d973-2496-491c-b8d0-bc0558a31020-openshift-service-ca\") pod \"maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:45.435318 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:46:45.435242    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0523d973-2496-491c-b8d0-bc0558a31020-maas-api-tls podName:0523d973-2496-491c-b8d0-bc0558a31020 nodeName:}" failed. No retries permitted until 2026-07-24 12:46:45.935223277 +0000 UTC m=+1679.462970311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0523d973-2496-491c-b8d0-bc0558a31020-maas-api-tls") pod "maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c" (UID: "0523d973-2496-491c-b8d0-bc0558a31020") : secret "maas-api-e2e-ait-52688321-serving-cert" not found
Jul 24 12:46:45.435916 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.435898    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0523d973-2496-491c-b8d0-bc0558a31020-openshift-service-ca\") pod \"maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:45.460369 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.460338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s27m8\" (UniqueName: \"kubernetes.io/projected/0523d973-2496-491c-b8d0-bc0558a31020-kube-api-access-s27m8\") pod \"maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:45.669288 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.669197    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:45.939936 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.939838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0523d973-2496-491c-b8d0-bc0558a31020-maas-api-tls\") pod \"maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:45.942554 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:45.942528    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0523d973-2496-491c-b8d0-bc0558a31020-maas-api-tls\") pod \"maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:46.160324 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.160273    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:46.171285 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.171259    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:46.270467 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.270434    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:46:46.353190 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.353155    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"]
Jul 24 12:46:46.356840 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.356824    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"
Jul 24 12:46:46.360000 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.359973    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-52688321\""
Jul 24 12:46:46.360226 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.360209    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-52688321-dockercfg-mm6jl\""
Jul 24 12:46:46.371832 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.371801    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"]
Jul 24 12:46:46.445488 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.445454    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6de5d91-6904-4ab9-8787-5e10bae78fdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj\" (UID: \"b6de5d91-6904-4ab9-8787-5e10bae78fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"
Jul 24 12:46:46.445664 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.445501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t545f\" (UniqueName: \"kubernetes.io/projected/b6de5d91-6904-4ab9-8787-5e10bae78fdb-kube-api-access-t545f\") pod \"payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj\" (UID: \"b6de5d91-6904-4ab9-8787-5e10bae78fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"
Jul 24 12:46:46.471708 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.471675    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"]
Jul 24 12:46:46.475316 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.475289    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"
Jul 24 12:46:46.488210 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.488180    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"]
Jul 24 12:46:46.532646 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.532615    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"]
Jul 24 12:46:46.533164 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:46:46.533135    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0523d973_2496_491c_b8d0_bc0558a31020.slice/crio-25baaa07fb2ce7283a471abe540bf36dd769085d6a2b91157148b045413a544e WatchSource:0}: Error finding container 25baaa07fb2ce7283a471abe540bf36dd769085d6a2b91157148b045413a544e: Status 404 returned error can't find the container with id 25baaa07fb2ce7283a471abe540bf36dd769085d6a2b91157148b045413a544e
Jul 24 12:46:46.547037 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.546994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6de5d91-6904-4ab9-8787-5e10bae78fdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj\" (UID: \"b6de5d91-6904-4ab9-8787-5e10bae78fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"
Jul 24 12:46:46.547160 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.547058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t545f\" (UniqueName: \"kubernetes.io/projected/b6de5d91-6904-4ab9-8787-5e10bae78fdb-kube-api-access-t545f\") pod \"payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj\" (UID: \"b6de5d91-6904-4ab9-8787-5e10bae78fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"
Jul 24 12:46:46.547686 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.547666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6de5d91-6904-4ab9-8787-5e10bae78fdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj\" (UID: \"b6de5d91-6904-4ab9-8787-5e10bae78fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"
Jul 24 12:46:46.558629 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.558600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t545f\" (UniqueName: \"kubernetes.io/projected/b6de5d91-6904-4ab9-8787-5e10bae78fdb-kube-api-access-t545f\") pod \"payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj\" (UID: \"b6de5d91-6904-4ab9-8787-5e10bae78fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"
Jul 24 12:46:46.648287 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.648247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-267gq\" (UniqueName: \"kubernetes.io/projected/f9ed496e-81ab-46ba-af25-ca3922fe5392-kube-api-access-267gq\") pod \"payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t\" (UID: \"f9ed496e-81ab-46ba-af25-ca3922fe5392\") " pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"
Jul 24 12:46:46.648482 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.648376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ed496e-81ab-46ba-af25-ca3922fe5392-plugins-config-volume\") pod \"payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t\" (UID: \"f9ed496e-81ab-46ba-af25-ca3922fe5392\") " pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"
Jul 24 12:46:46.668886 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.668848    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"
Jul 24 12:46:46.749353 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.749317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ed496e-81ab-46ba-af25-ca3922fe5392-plugins-config-volume\") pod \"payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t\" (UID: \"f9ed496e-81ab-46ba-af25-ca3922fe5392\") " pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"
Jul 24 12:46:46.749545 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.749370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-267gq\" (UniqueName: \"kubernetes.io/projected/f9ed496e-81ab-46ba-af25-ca3922fe5392-kube-api-access-267gq\") pod \"payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t\" (UID: \"f9ed496e-81ab-46ba-af25-ca3922fe5392\") " pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"
Jul 24 12:46:46.750055 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.750034    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ed496e-81ab-46ba-af25-ca3922fe5392-plugins-config-volume\") pod \"payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t\" (UID: \"f9ed496e-81ab-46ba-af25-ca3922fe5392\") " pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"
Jul 24 12:46:46.763540 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.762873    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-267gq\" (UniqueName: \"kubernetes.io/projected/f9ed496e-81ab-46ba-af25-ca3922fe5392-kube-api-access-267gq\") pod \"payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t\" (UID: \"f9ed496e-81ab-46ba-af25-ca3922fe5392\") " pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"
Jul 24 12:46:46.787974 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.787939    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"
Jul 24 12:46:46.827793 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.827759    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"]
Jul 24 12:46:46.828944 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:46:46.828917    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6de5d91_6904_4ab9_8787_5e10bae78fdb.slice/crio-7b4cb70967e25532d78d3103dd00c08d492d05477ba10dcfbc43a4d83a3f026b WatchSource:0}: Error finding container 7b4cb70967e25532d78d3103dd00c08d492d05477ba10dcfbc43a4d83a3f026b: Status 404 returned error can't find the container with id 7b4cb70967e25532d78d3103dd00c08d492d05477ba10dcfbc43a4d83a3f026b
Jul 24 12:46:46.948773 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.948744    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"]
Jul 24 12:46:46.956136 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:46:46.956103    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ed496e_81ab_46ba_af25_ca3922fe5392.slice/crio-1c4c934f2b5a5f014b0298f5d933985b5ff9a7dac5793b101054ee64110df34a WatchSource:0}: Error finding container 1c4c934f2b5a5f014b0298f5d933985b5ff9a7dac5793b101054ee64110df34a: Status 404 returned error can't find the container with id 1c4c934f2b5a5f014b0298f5d933985b5ff9a7dac5793b101054ee64110df34a
Jul 24 12:46:46.995008 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.994977    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t" event={"ID":"f9ed496e-81ab-46ba-af25-ca3922fe5392","Type":"ContainerStarted","Data":"1c4c934f2b5a5f014b0298f5d933985b5ff9a7dac5793b101054ee64110df34a"}
Jul 24 12:46:46.996830 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.996801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj" event={"ID":"b6de5d91-6904-4ab9-8787-5e10bae78fdb","Type":"ContainerStarted","Data":"0d8b500a890710af943abfb73ba275484be32d99edd4888fd5893f598426c7a2"}
Jul 24 12:46:46.996985 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.996967    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"
Jul 24 12:46:46.997084 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.997030    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj" event={"ID":"b6de5d91-6904-4ab9-8787-5e10bae78fdb","Type":"ContainerStarted","Data":"7b4cb70967e25532d78d3103dd00c08d492d05477ba10dcfbc43a4d83a3f026b"}
Jul 24 12:46:46.998716 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:46.998696    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c" event={"ID":"0523d973-2496-491c-b8d0-bc0558a31020","Type":"ContainerStarted","Data":"25baaa07fb2ce7283a471abe540bf36dd769085d6a2b91157148b045413a544e"}
Jul 24 12:46:47.027394 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:47.027336    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj" podStartSLOduration=1.027314571 podStartE2EDuration="1.027314571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:46:47.025326197 +0000 UTC m=+1680.553073252" watchObservedRunningTime="2026-07-24 12:46:47.027314571 +0000 UTC m=+1680.555061628"
Jul 24 12:46:48.005533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:48.005494    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c" event={"ID":"0523d973-2496-491c-b8d0-bc0558a31020","Type":"ContainerStarted","Data":"12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109"}
Jul 24 12:46:48.006057 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:48.005557    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:48.007044 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:48.006990    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t" event={"ID":"f9ed496e-81ab-46ba-af25-ca3922fe5392","Type":"ContainerStarted","Data":"b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae"}
Jul 24 12:46:48.027985 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:48.027921    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c" podStartSLOduration=2.6341939439999997 podStartE2EDuration="3.027901637s" totalImagesPullingTime="393.707693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:46:48.024733745 +0000 UTC m=+1681.552480801" watchObservedRunningTime="2026-07-24 12:46:48.027901637 +0000 UTC m=+1681.555648691"
Jul 24 12:46:48.044470 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:48.044421    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t" podStartSLOduration=2.044406011 podStartE2EDuration="2.044406011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:46:48.042515374 +0000 UTC m=+1681.570262429" watchObservedRunningTime="2026-07-24 12:46:48.044406011 +0000 UTC m=+1681.572153116"
Jul 24 12:46:49.012474 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:49.012441    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"
Jul 24 12:46:54.017741 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:54.017709    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:46:58.009504 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:46:58.009457    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj" podUID="b6de5d91-6904-4ab9-8787-5e10bae78fdb" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.49:9004: connect: connection refused"
Jul 24 12:47:00.019871 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:00.019841    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"
Jul 24 12:47:08.008086 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:08.007975    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"
Jul 24 12:47:21.783709 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:21.783673    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"]
Jul 24 12:47:21.784308 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:21.783938    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c" podUID="0523d973-2496-491c-b8d0-bc0558a31020" containerName="maas-api" containerID="cri-o://12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109" gracePeriod=30
Jul 24 12:47:21.820958 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:21.820914    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:21.858136 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:21.858102    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"]
Jul 24 12:47:21.858354 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:21.858323    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t" podUID="f9ed496e-81ab-46ba-af25-ca3922fe5392" containerName="payload-processing" containerID="cri-o://b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae" gracePeriod=30
Jul 24 12:47:21.868441 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:21.868209    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"]
Jul 24 12:47:21.868533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:21.868442    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj" podUID="b6de5d91-6904-4ab9-8787-5e10bae78fdb" containerName="payload-pre-processing" containerID="cri-o://0d8b500a890710af943abfb73ba275484be32d99edd4888fd5893f598426c7a2" gracePeriod=30
Jul 24 12:47:21.914619 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:21.914562    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:22.002104 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.002059    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:22.143521 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.143490    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:47:22.184092 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.184060    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"
Jul 24 12:47:22.193942 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.193913    2566 generic.go:356] "Generic (PLEG): container finished" podID="0523d973-2496-491c-b8d0-bc0558a31020" containerID="12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109" exitCode=0
Jul 24 12:47:22.194162 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.193949    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c" event={"ID":"0523d973-2496-491c-b8d0-bc0558a31020","Type":"ContainerDied","Data":"12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109"}
Jul 24 12:47:22.194162 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.193991    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"
Jul 24 12:47:22.194162 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.194076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c" event={"ID":"0523d973-2496-491c-b8d0-bc0558a31020","Type":"ContainerDied","Data":"25baaa07fb2ce7283a471abe540bf36dd769085d6a2b91157148b045413a544e"}
Jul 24 12:47:22.194162 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.194111    2566 scope.go:117] "RemoveContainer" containerID="12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109"
Jul 24 12:47:22.194371 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.194353    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"
Jul 24 12:47:22.195388 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.195366    2566 generic.go:356] "Generic (PLEG): container finished" podID="f9ed496e-81ab-46ba-af25-ca3922fe5392" containerID="b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae" exitCode=0
Jul 24 12:47:22.195511 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.195409    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"
Jul 24 12:47:22.195511 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.195453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t" event={"ID":"f9ed496e-81ab-46ba-af25-ca3922fe5392","Type":"ContainerDied","Data":"b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae"}
Jul 24 12:47:22.195511 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.195506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t" event={"ID":"f9ed496e-81ab-46ba-af25-ca3922fe5392","Type":"ContainerDied","Data":"1c4c934f2b5a5f014b0298f5d933985b5ff9a7dac5793b101054ee64110df34a"}
Jul 24 12:47:22.196774 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.196750    2566 generic.go:356] "Generic (PLEG): container finished" podID="b6de5d91-6904-4ab9-8787-5e10bae78fdb" containerID="0d8b500a890710af943abfb73ba275484be32d99edd4888fd5893f598426c7a2" exitCode=0
Jul 24 12:47:22.196880 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.196780    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"
Jul 24 12:47:22.196880 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.196827    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj" event={"ID":"b6de5d91-6904-4ab9-8787-5e10bae78fdb","Type":"ContainerDied","Data":"0d8b500a890710af943abfb73ba275484be32d99edd4888fd5893f598426c7a2"}
Jul 24 12:47:22.205131 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.205003    2566 scope.go:117] "RemoveContainer" containerID="12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109"
Jul 24 12:47:22.205583 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:47:22.205548    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109\": container with ID starting with 12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109 not found: ID does not exist" containerID="12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109"
Jul 24 12:47:22.205697 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.205587    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109"} err="failed to get container status \"12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109\": rpc error: code = NotFound desc = could not find container \"12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109\": container with ID starting with 12178b834959c79e394d6b179300683149882a14d9e75ca108cada9db800a109 not found: ID does not exist"
Jul 24 12:47:22.205697 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.205608    2566 scope.go:117] "RemoveContainer" containerID="b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae"
Jul 24 12:47:22.217081 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.217059    2566 scope.go:117] "RemoveContainer" containerID="b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae"
Jul 24 12:47:22.217445 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:47:22.217424    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae\": container with ID starting with b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae not found: ID does not exist" containerID="b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae"
Jul 24 12:47:22.217549 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.217455    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae"} err="failed to get container status \"b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae\": rpc error: code = NotFound desc = could not find container \"b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae\": container with ID starting with b7224ad75124963eedd5eab2bb885a8f70c98f7d0e3b29b9dd821880d04abfae not found: ID does not exist"
Jul 24 12:47:22.217549 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.217472    2566 scope.go:117] "RemoveContainer" containerID="0d8b500a890710af943abfb73ba275484be32d99edd4888fd5893f598426c7a2"
Jul 24 12:47:22.287194 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.287158    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-267gq\" (UniqueName: \"kubernetes.io/projected/f9ed496e-81ab-46ba-af25-ca3922fe5392-kube-api-access-267gq\") pod \"f9ed496e-81ab-46ba-af25-ca3922fe5392\" (UID: \"f9ed496e-81ab-46ba-af25-ca3922fe5392\") "
Jul 24 12:47:22.287194 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.287197    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ed496e-81ab-46ba-af25-ca3922fe5392-plugins-config-volume\") pod \"f9ed496e-81ab-46ba-af25-ca3922fe5392\" (UID: \"f9ed496e-81ab-46ba-af25-ca3922fe5392\") "
Jul 24 12:47:22.287476 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.287232    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s27m8\" (UniqueName: \"kubernetes.io/projected/0523d973-2496-491c-b8d0-bc0558a31020-kube-api-access-s27m8\") pod \"0523d973-2496-491c-b8d0-bc0558a31020\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") "
Jul 24 12:47:22.287476 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.287255    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6de5d91-6904-4ab9-8787-5e10bae78fdb-plugins-config-volume\") pod \"b6de5d91-6904-4ab9-8787-5e10bae78fdb\" (UID: \"b6de5d91-6904-4ab9-8787-5e10bae78fdb\") "
Jul 24 12:47:22.287476 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.287363    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t545f\" (UniqueName: \"kubernetes.io/projected/b6de5d91-6904-4ab9-8787-5e10bae78fdb-kube-api-access-t545f\") pod \"b6de5d91-6904-4ab9-8787-5e10bae78fdb\" (UID: \"b6de5d91-6904-4ab9-8787-5e10bae78fdb\") "
Jul 24 12:47:22.287476 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.287394    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0523d973-2496-491c-b8d0-bc0558a31020-maas-api-tls\") pod \"0523d973-2496-491c-b8d0-bc0558a31020\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") "
Jul 24 12:47:22.287476 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.287445    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0523d973-2496-491c-b8d0-bc0558a31020-openshift-service-ca\") pod \"0523d973-2496-491c-b8d0-bc0558a31020\" (UID: \"0523d973-2496-491c-b8d0-bc0558a31020\") "
Jul 24 12:47:22.287747 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.287662    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ed496e-81ab-46ba-af25-ca3922fe5392-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9ed496e-81ab-46ba-af25-ca3922fe5392" (UID: "f9ed496e-81ab-46ba-af25-ca3922fe5392"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:47:22.287747 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.287721    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6de5d91-6904-4ab9-8787-5e10bae78fdb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6de5d91-6904-4ab9-8787-5e10bae78fdb" (UID: "b6de5d91-6904-4ab9-8787-5e10bae78fdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:47:22.288006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.287951    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ed496e-81ab-46ba-af25-ca3922fe5392-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:47:22.288006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.287979    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6de5d91-6904-4ab9-8787-5e10bae78fdb-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:47:22.288006 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.287947    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0523d973-2496-491c-b8d0-bc0558a31020-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0523d973-2496-491c-b8d0-bc0558a31020" (UID: "0523d973-2496-491c-b8d0-bc0558a31020"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:47:22.289717 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.289684    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0523d973-2496-491c-b8d0-bc0558a31020-kube-api-access-s27m8" (OuterVolumeSpecName: "kube-api-access-s27m8") pod "0523d973-2496-491c-b8d0-bc0558a31020" (UID: "0523d973-2496-491c-b8d0-bc0558a31020"). InnerVolumeSpecName "kube-api-access-s27m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:47:22.289825 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.289745    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6de5d91-6904-4ab9-8787-5e10bae78fdb-kube-api-access-t545f" (OuterVolumeSpecName: "kube-api-access-t545f") pod "b6de5d91-6904-4ab9-8787-5e10bae78fdb" (UID: "b6de5d91-6904-4ab9-8787-5e10bae78fdb"). InnerVolumeSpecName "kube-api-access-t545f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:47:22.289874 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.289834    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ed496e-81ab-46ba-af25-ca3922fe5392-kube-api-access-267gq" (OuterVolumeSpecName: "kube-api-access-267gq") pod "f9ed496e-81ab-46ba-af25-ca3922fe5392" (UID: "f9ed496e-81ab-46ba-af25-ca3922fe5392"). InnerVolumeSpecName "kube-api-access-267gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:47:22.290137 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.290106    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0523d973-2496-491c-b8d0-bc0558a31020-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0523d973-2496-491c-b8d0-bc0558a31020" (UID: "0523d973-2496-491c-b8d0-bc0558a31020"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:47:22.388544 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.388506    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-267gq\" (UniqueName: \"kubernetes.io/projected/f9ed496e-81ab-46ba-af25-ca3922fe5392-kube-api-access-267gq\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:47:22.388544 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.388541    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s27m8\" (UniqueName: \"kubernetes.io/projected/0523d973-2496-491c-b8d0-bc0558a31020-kube-api-access-s27m8\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:47:22.388544 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.388550    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t545f\" (UniqueName: \"kubernetes.io/projected/b6de5d91-6904-4ab9-8787-5e10bae78fdb-kube-api-access-t545f\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:47:22.388789 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.388560    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0523d973-2496-491c-b8d0-bc0558a31020-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:47:22.388789 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.388569    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0523d973-2496-491c-b8d0-bc0558a31020-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:47:22.520265 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.520226    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"]
Jul 24 12:47:22.526366 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.526334    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-52688321-6cbc9cb47-8hp8c"]
Jul 24 12:47:22.543416 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.543314    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"]
Jul 24 12:47:22.550609 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.550580    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-52688321-7ff5dfd5cd-gdk8t"]
Jul 24 12:47:22.564724 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.564692    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"]
Jul 24 12:47:22.570236 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.570207    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-52688321-5d9fdd448-pbckj"]
Jul 24 12:47:22.975323 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.975285    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0523d973-2496-491c-b8d0-bc0558a31020" path="/var/lib/kubelet/pods/0523d973-2496-491c-b8d0-bc0558a31020/volumes"
Jul 24 12:47:22.975694 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.975645    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6de5d91-6904-4ab9-8787-5e10bae78fdb" path="/var/lib/kubelet/pods/b6de5d91-6904-4ab9-8787-5e10bae78fdb/volumes"
Jul 24 12:47:22.975947 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:22.975935    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9ed496e-81ab-46ba-af25-ca3922fe5392" path="/var/lib/kubelet/pods/f9ed496e-81ab-46ba-af25-ca3922fe5392/volumes"
Jul 24 12:47:26.272983 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:26.272939    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:27.560217 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:27.560180    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:27.793167 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:27.793113    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:28.029690 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:28.029652    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:28.347572 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:28.347490    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:28.582627 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:28.582589    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:31.240742 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.240708    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:31.332098 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.332063    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:31.333728 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.333700    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"]
Jul 24 12:47:31.334330 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.334305    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ed496e-81ab-46ba-af25-ca3922fe5392" containerName="payload-processing"
Jul 24 12:47:31.334330 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.334331    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ed496e-81ab-46ba-af25-ca3922fe5392" containerName="payload-processing"
Jul 24 12:47:31.334499 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.334348    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6de5d91-6904-4ab9-8787-5e10bae78fdb" containerName="payload-pre-processing"
Jul 24 12:47:31.334499 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.334358    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6de5d91-6904-4ab9-8787-5e10bae78fdb" containerName="payload-pre-processing"
Jul 24 12:47:31.334499 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.334377    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0523d973-2496-491c-b8d0-bc0558a31020" containerName="maas-api"
Jul 24 12:47:31.334499 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.334385    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0523d973-2496-491c-b8d0-bc0558a31020" containerName="maas-api"
Jul 24 12:47:31.334499 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.334490    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6de5d91-6904-4ab9-8787-5e10bae78fdb" containerName="payload-pre-processing"
Jul 24 12:47:31.334499 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.334502    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ed496e-81ab-46ba-af25-ca3922fe5392" containerName="payload-processing"
Jul 24 12:47:31.334778 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.334512    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0523d973-2496-491c-b8d0-bc0558a31020" containerName="maas-api"
Jul 24 12:47:31.339298 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.339279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:31.342584 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.342567    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-266bd0de-serving-cert\""
Jul 24 12:47:31.350432 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.350408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"]
Jul 24 12:47:31.418348 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.418318    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:31.476157 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.476125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4vg\" (UniqueName: \"kubernetes.io/projected/ae7b583c-1c06-497b-8763-af16feb44d1e-kube-api-access-7x4vg\") pod \"maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:31.476324 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.476178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7b583c-1c06-497b-8763-af16feb44d1e-openshift-service-ca\") pod \"maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:31.476324 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.476209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7b583c-1c06-497b-8763-af16feb44d1e-maas-api-tls\") pod \"maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:31.577683 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.577657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4vg\" (UniqueName: \"kubernetes.io/projected/ae7b583c-1c06-497b-8763-af16feb44d1e-kube-api-access-7x4vg\") pod \"maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:31.577853 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.577702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7b583c-1c06-497b-8763-af16feb44d1e-openshift-service-ca\") pod \"maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:31.577853 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.577724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7b583c-1c06-497b-8763-af16feb44d1e-maas-api-tls\") pod \"maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:31.577853 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:47:31.577846    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-serving-cert: secret "maas-api-e2e-derive-266bd0de-serving-cert" not found
Jul 24 12:47:31.577983 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:47:31.577913    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7b583c-1c06-497b-8763-af16feb44d1e-maas-api-tls podName:ae7b583c-1c06-497b-8763-af16feb44d1e nodeName:}" failed. No retries permitted until 2026-07-24 12:47:32.077892698 +0000 UTC m=+1725.605639752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae7b583c-1c06-497b-8763-af16feb44d1e-maas-api-tls") pod "maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c" (UID: "ae7b583c-1c06-497b-8763-af16feb44d1e") : secret "maas-api-e2e-derive-266bd0de-serving-cert" not found
Jul 24 12:47:31.578417 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.578398    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7b583c-1c06-497b-8763-af16feb44d1e-openshift-service-ca\") pod \"maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:31.589483 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.589461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4vg\" (UniqueName: \"kubernetes.io/projected/ae7b583c-1c06-497b-8763-af16feb44d1e-kube-api-access-7x4vg\") pod \"maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:31.674795 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.674765    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:31.943288 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:31.943191    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:32.042333 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.042298    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:47:32.081283 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.081247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7b583c-1c06-497b-8763-af16feb44d1e-maas-api-tls\") pod \"maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:32.088928 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.085293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7b583c-1c06-497b-8763-af16feb44d1e-maas-api-tls\") pod \"maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:32.104801 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.104777    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"]
Jul 24 12:47:32.108321 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.108304    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"
Jul 24 12:47:32.111879 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.111860    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-266bd0de-dockercfg-xl99d\""
Jul 24 12:47:32.112699 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.112686    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-266bd0de\""
Jul 24 12:47:32.129217 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.129194    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"]
Jul 24 12:47:32.149371 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.149345    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"]
Jul 24 12:47:32.152822 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.152800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"
Jul 24 12:47:32.174971 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.174948    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"]
Jul 24 12:47:32.250528 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.250500    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:32.282604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.282571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd88b50a-4759-414c-88c1-8a159dc89560-plugins-config-volume\") pod \"payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7\" (UID: \"dd88b50a-4759-414c-88c1-8a159dc89560\") " pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"
Jul 24 12:47:32.282758 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.282628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c817bf-546e-4446-abd4-2db4c51c0e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4\" (UID: \"29c817bf-546e-4446-abd4-2db4c51c0e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"
Jul 24 12:47:32.282758 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.282650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxbh\" (UniqueName: \"kubernetes.io/projected/29c817bf-546e-4446-abd4-2db4c51c0e5b-kube-api-access-4bxbh\") pod \"payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4\" (UID: \"29c817bf-546e-4446-abd4-2db4c51c0e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"
Jul 24 12:47:32.282758 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.282740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmphw\" (UniqueName: \"kubernetes.io/projected/dd88b50a-4759-414c-88c1-8a159dc89560-kube-api-access-pmphw\") pod \"payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7\" (UID: \"dd88b50a-4759-414c-88c1-8a159dc89560\") " pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"
Jul 24 12:47:32.383490 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.383458    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c817bf-546e-4446-abd4-2db4c51c0e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4\" (UID: \"29c817bf-546e-4446-abd4-2db4c51c0e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"
Jul 24 12:47:32.383490 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.383494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxbh\" (UniqueName: \"kubernetes.io/projected/29c817bf-546e-4446-abd4-2db4c51c0e5b-kube-api-access-4bxbh\") pod \"payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4\" (UID: \"29c817bf-546e-4446-abd4-2db4c51c0e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"
Jul 24 12:47:32.383701 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.383513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmphw\" (UniqueName: \"kubernetes.io/projected/dd88b50a-4759-414c-88c1-8a159dc89560-kube-api-access-pmphw\") pod \"payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7\" (UID: \"dd88b50a-4759-414c-88c1-8a159dc89560\") " pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"
Jul 24 12:47:32.383701 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.383614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd88b50a-4759-414c-88c1-8a159dc89560-plugins-config-volume\") pod \"payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7\" (UID: \"dd88b50a-4759-414c-88c1-8a159dc89560\") " pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"
Jul 24 12:47:32.384415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.384390    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c817bf-546e-4446-abd4-2db4c51c0e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4\" (UID: \"29c817bf-546e-4446-abd4-2db4c51c0e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"
Jul 24 12:47:32.384519 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.384397    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd88b50a-4759-414c-88c1-8a159dc89560-plugins-config-volume\") pod \"payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7\" (UID: \"dd88b50a-4759-414c-88c1-8a159dc89560\") " pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"
Jul 24 12:47:32.392936 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.392912    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"]
Jul 24 12:47:32.395098 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:47:32.395069    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7b583c_1c06_497b_8763_af16feb44d1e.slice/crio-a241b85550978dbca12c79d33573c137921c0e9e24b778c057ef22daf96fcf6b WatchSource:0}: Error finding container a241b85550978dbca12c79d33573c137921c0e9e24b778c057ef22daf96fcf6b: Status 404 returned error can't find the container with id a241b85550978dbca12c79d33573c137921c0e9e24b778c057ef22daf96fcf6b
Jul 24 12:47:32.395504 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.395485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmphw\" (UniqueName: \"kubernetes.io/projected/dd88b50a-4759-414c-88c1-8a159dc89560-kube-api-access-pmphw\") pod \"payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7\" (UID: \"dd88b50a-4759-414c-88c1-8a159dc89560\") " pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"
Jul 24 12:47:32.395587 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.395483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxbh\" (UniqueName: \"kubernetes.io/projected/29c817bf-546e-4446-abd4-2db4c51c0e5b-kube-api-access-4bxbh\") pod \"payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4\" (UID: \"29c817bf-546e-4446-abd4-2db4c51c0e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"
Jul 24 12:47:32.418951 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.418930    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"
Jul 24 12:47:32.464135 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.464094    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"
Jul 24 12:47:32.576828 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.576766    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"]
Jul 24 12:47:32.578954 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:47:32.578914    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd88b50a_4759_414c_88c1_8a159dc89560.slice/crio-afc9268c2bdd00a8c270846d6b5c1f90c65841dc029e43f3b37f07680c3af031 WatchSource:0}: Error finding container afc9268c2bdd00a8c270846d6b5c1f90c65841dc029e43f3b37f07680c3af031: Status 404 returned error can't find the container with id afc9268c2bdd00a8c270846d6b5c1f90c65841dc029e43f3b37f07680c3af031
Jul 24 12:47:32.627299 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:32.627278    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"]
Jul 24 12:47:32.630092 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:47:32.630067    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c817bf_546e_4446_abd4_2db4c51c0e5b.slice/crio-66a831c35523665ff0d1242dee9b31187797d5e14bde4e3d0f9d0209265fe53d WatchSource:0}: Error finding container 66a831c35523665ff0d1242dee9b31187797d5e14bde4e3d0f9d0209265fe53d: Status 404 returned error can't find the container with id 66a831c35523665ff0d1242dee9b31187797d5e14bde4e3d0f9d0209265fe53d
Jul 24 12:47:33.255867 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:33.255834    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c" event={"ID":"ae7b583c-1c06-497b-8763-af16feb44d1e","Type":"ContainerStarted","Data":"4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e"}
Jul 24 12:47:33.256299 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:33.255892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c" event={"ID":"ae7b583c-1c06-497b-8763-af16feb44d1e","Type":"ContainerStarted","Data":"a241b85550978dbca12c79d33573c137921c0e9e24b778c057ef22daf96fcf6b"}
Jul 24 12:47:33.256299 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:33.255922    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:33.257350 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:33.257322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7" event={"ID":"dd88b50a-4759-414c-88c1-8a159dc89560","Type":"ContainerStarted","Data":"fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c"}
Jul 24 12:47:33.257480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:33.257369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7" event={"ID":"dd88b50a-4759-414c-88c1-8a159dc89560","Type":"ContainerStarted","Data":"afc9268c2bdd00a8c270846d6b5c1f90c65841dc029e43f3b37f07680c3af031"}
Jul 24 12:47:33.257480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:33.257413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"
Jul 24 12:47:33.258712 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:33.258692    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4" event={"ID":"29c817bf-546e-4446-abd4-2db4c51c0e5b","Type":"ContainerStarted","Data":"4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5"}
Jul 24 12:47:33.258805 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:33.258725    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4" event={"ID":"29c817bf-546e-4446-abd4-2db4c51c0e5b","Type":"ContainerStarted","Data":"66a831c35523665ff0d1242dee9b31187797d5e14bde4e3d0f9d0209265fe53d"}
Jul 24 12:47:33.258805 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:33.258744    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"
Jul 24 12:47:33.276390 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:33.276326    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c" podStartSLOduration=1.844850336 podStartE2EDuration="2.276313725s" totalImagesPullingTime="431.463389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:47:33.273897074 +0000 UTC m=+1726.801644152" watchObservedRunningTime="2026-07-24 12:47:33.276313725 +0000 UTC m=+1726.804060779"
Jul 24 12:47:33.294280 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:33.294241    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7" podStartSLOduration=1.294229073 podStartE2EDuration="1.294229073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:47:33.292122459 +0000 UTC m=+1726.819869513" watchObservedRunningTime="2026-07-24 12:47:33.294229073 +0000 UTC m=+1726.821976197"
Jul 24 12:47:33.310783 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:33.310737    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4" podStartSLOduration=1.310721716 podStartE2EDuration="1.310721716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:47:33.308809437 +0000 UTC m=+1726.836556492" watchObservedRunningTime="2026-07-24 12:47:33.310721716 +0000 UTC m=+1726.838468776"
Jul 24 12:47:39.269427 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:39.269398    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:47:44.265352 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:44.265321    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"
Jul 24 12:47:44.265838 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:47:44.265402    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"
Jul 24 12:48:06.846712 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:06.846669    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"]
Jul 24 12:48:06.847325 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:06.847201    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c" podUID="ae7b583c-1c06-497b-8763-af16feb44d1e" containerName="maas-api" containerID="cri-o://4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e" gracePeriod=30
Jul 24 12:48:06.873265 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:06.873232    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:06.907115 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:06.907082    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"]
Jul 24 12:48:06.907406 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:06.907364    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7" podUID="dd88b50a-4759-414c-88c1-8a159dc89560" containerName="payload-processing" containerID="cri-o://fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c" gracePeriod=30
Jul 24 12:48:06.913305 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:06.913278    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"]
Jul 24 12:48:06.913843 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:06.913789    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4" podUID="29c817bf-546e-4446-abd4-2db4c51c0e5b" containerName="payload-pre-processing" containerID="cri-o://4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5" gracePeriod=30
Jul 24 12:48:06.958001 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:06.957966    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:07.033363 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.033275    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:07.202502 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.202482    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"
Jul 24 12:48:07.205864 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.205839    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"
Jul 24 12:48:07.209326 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.209304    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:48:07.311693 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.311664    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x4vg\" (UniqueName: \"kubernetes.io/projected/ae7b583c-1c06-497b-8763-af16feb44d1e-kube-api-access-7x4vg\") pod \"ae7b583c-1c06-497b-8763-af16feb44d1e\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") "
Jul 24 12:48:07.311693 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.311694    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7b583c-1c06-497b-8763-af16feb44d1e-openshift-service-ca\") pod \"ae7b583c-1c06-497b-8763-af16feb44d1e\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") "
Jul 24 12:48:07.311930 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.311767    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7b583c-1c06-497b-8763-af16feb44d1e-maas-api-tls\") pod \"ae7b583c-1c06-497b-8763-af16feb44d1e\" (UID: \"ae7b583c-1c06-497b-8763-af16feb44d1e\") "
Jul 24 12:48:07.311930 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.311784    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmphw\" (UniqueName: \"kubernetes.io/projected/dd88b50a-4759-414c-88c1-8a159dc89560-kube-api-access-pmphw\") pod \"dd88b50a-4759-414c-88c1-8a159dc89560\" (UID: \"dd88b50a-4759-414c-88c1-8a159dc89560\") "
Jul 24 12:48:07.311930 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.311816    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd88b50a-4759-414c-88c1-8a159dc89560-plugins-config-volume\") pod \"dd88b50a-4759-414c-88c1-8a159dc89560\" (UID: \"dd88b50a-4759-414c-88c1-8a159dc89560\") "
Jul 24 12:48:07.312114 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.311965    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c817bf-546e-4446-abd4-2db4c51c0e5b-plugins-config-volume\") pod \"29c817bf-546e-4446-abd4-2db4c51c0e5b\" (UID: \"29c817bf-546e-4446-abd4-2db4c51c0e5b\") "
Jul 24 12:48:07.312114 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.312051    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bxbh\" (UniqueName: \"kubernetes.io/projected/29c817bf-546e-4446-abd4-2db4c51c0e5b-kube-api-access-4bxbh\") pod \"29c817bf-546e-4446-abd4-2db4c51c0e5b\" (UID: \"29c817bf-546e-4446-abd4-2db4c51c0e5b\") "
Jul 24 12:48:07.312308 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.312282    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7b583c-1c06-497b-8763-af16feb44d1e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae7b583c-1c06-497b-8763-af16feb44d1e" (UID: "ae7b583c-1c06-497b-8763-af16feb44d1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:48:07.312452 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.312318    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd88b50a-4759-414c-88c1-8a159dc89560-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd88b50a-4759-414c-88c1-8a159dc89560" (UID: "dd88b50a-4759-414c-88c1-8a159dc89560"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:48:07.312452 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.312403    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c817bf-546e-4446-abd4-2db4c51c0e5b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29c817bf-546e-4446-abd4-2db4c51c0e5b" (UID: "29c817bf-546e-4446-abd4-2db4c51c0e5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:48:07.314206 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.314182    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7b583c-1c06-497b-8763-af16feb44d1e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae7b583c-1c06-497b-8763-af16feb44d1e" (UID: "ae7b583c-1c06-497b-8763-af16feb44d1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:48:07.314532 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.314513    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd88b50a-4759-414c-88c1-8a159dc89560-kube-api-access-pmphw" (OuterVolumeSpecName: "kube-api-access-pmphw") pod "dd88b50a-4759-414c-88c1-8a159dc89560" (UID: "dd88b50a-4759-414c-88c1-8a159dc89560"). InnerVolumeSpecName "kube-api-access-pmphw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:48:07.314606 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.314526    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c817bf-546e-4446-abd4-2db4c51c0e5b-kube-api-access-4bxbh" (OuterVolumeSpecName: "kube-api-access-4bxbh") pod "29c817bf-546e-4446-abd4-2db4c51c0e5b" (UID: "29c817bf-546e-4446-abd4-2db4c51c0e5b"). InnerVolumeSpecName "kube-api-access-4bxbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:48:07.314606 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.314539    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7b583c-1c06-497b-8763-af16feb44d1e-kube-api-access-7x4vg" (OuterVolumeSpecName: "kube-api-access-7x4vg") pod "ae7b583c-1c06-497b-8763-af16feb44d1e" (UID: "ae7b583c-1c06-497b-8763-af16feb44d1e"). InnerVolumeSpecName "kube-api-access-7x4vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:48:07.413520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.413429    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7b583c-1c06-497b-8763-af16feb44d1e-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:48:07.413520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.413458    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmphw\" (UniqueName: \"kubernetes.io/projected/dd88b50a-4759-414c-88c1-8a159dc89560-kube-api-access-pmphw\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:48:07.413520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.413472    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd88b50a-4759-414c-88c1-8a159dc89560-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:48:07.413520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.413484    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c817bf-546e-4446-abd4-2db4c51c0e5b-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:48:07.413520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.413497    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bxbh\" (UniqueName: \"kubernetes.io/projected/29c817bf-546e-4446-abd4-2db4c51c0e5b-kube-api-access-4bxbh\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:48:07.413520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.413510    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x4vg\" (UniqueName: \"kubernetes.io/projected/ae7b583c-1c06-497b-8763-af16feb44d1e-kube-api-access-7x4vg\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:48:07.413520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.413522    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7b583c-1c06-497b-8763-af16feb44d1e-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:48:07.435700 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.435667    2566 generic.go:356] "Generic (PLEG): container finished" podID="29c817bf-546e-4446-abd4-2db4c51c0e5b" containerID="4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5" exitCode=0
Jul 24 12:48:07.435900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.435733    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"
Jul 24 12:48:07.435900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.435756    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4" event={"ID":"29c817bf-546e-4446-abd4-2db4c51c0e5b","Type":"ContainerDied","Data":"4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5"}
Jul 24 12:48:07.435900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.435814    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4" event={"ID":"29c817bf-546e-4446-abd4-2db4c51c0e5b","Type":"ContainerDied","Data":"66a831c35523665ff0d1242dee9b31187797d5e14bde4e3d0f9d0209265fe53d"}
Jul 24 12:48:07.435900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.435830    2566 scope.go:117] "RemoveContainer" containerID="4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5"
Jul 24 12:48:07.437395 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.437372    2566 generic.go:356] "Generic (PLEG): container finished" podID="ae7b583c-1c06-497b-8763-af16feb44d1e" containerID="4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e" exitCode=0
Jul 24 12:48:07.437515 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.437402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c" event={"ID":"ae7b583c-1c06-497b-8763-af16feb44d1e","Type":"ContainerDied","Data":"4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e"}
Jul 24 12:48:07.437515 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.437419    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"
Jul 24 12:48:07.437515 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.437461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c" event={"ID":"ae7b583c-1c06-497b-8763-af16feb44d1e","Type":"ContainerDied","Data":"a241b85550978dbca12c79d33573c137921c0e9e24b778c057ef22daf96fcf6b"}
Jul 24 12:48:07.439072 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.439047    2566 generic.go:356] "Generic (PLEG): container finished" podID="dd88b50a-4759-414c-88c1-8a159dc89560" containerID="fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c" exitCode=0
Jul 24 12:48:07.439198 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.439111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7" event={"ID":"dd88b50a-4759-414c-88c1-8a159dc89560","Type":"ContainerDied","Data":"fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c"}
Jul 24 12:48:07.439198 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.439148    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"
Jul 24 12:48:07.439198 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.439154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7" event={"ID":"dd88b50a-4759-414c-88c1-8a159dc89560","Type":"ContainerDied","Data":"afc9268c2bdd00a8c270846d6b5c1f90c65841dc029e43f3b37f07680c3af031"}
Jul 24 12:48:07.446964 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.446816    2566 scope.go:117] "RemoveContainer" containerID="4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5"
Jul 24 12:48:07.447208 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:48:07.447184    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5\": container with ID starting with 4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5 not found: ID does not exist" containerID="4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5"
Jul 24 12:48:07.447294 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.447220    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5"} err="failed to get container status \"4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5\": rpc error: code = NotFound desc = could not find container \"4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5\": container with ID starting with 4d180138c1408ebd357bed386d5392f8ef0993207a3baf2cc5542d145fee8bb5 not found: ID does not exist"
Jul 24 12:48:07.447294 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.447240    2566 scope.go:117] "RemoveContainer" containerID="4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e"
Jul 24 12:48:07.456666 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.456595    2566 scope.go:117] "RemoveContainer" containerID="4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e"
Jul 24 12:48:07.456882 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:48:07.456863    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e\": container with ID starting with 4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e not found: ID does not exist" containerID="4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e"
Jul 24 12:48:07.456919 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.456889    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e"} err="failed to get container status \"4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e\": rpc error: code = NotFound desc = could not find container \"4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e\": container with ID starting with 4c6f91e92c36526c8507f013a6f89a034979f4cf155b015c5ad67d992809766e not found: ID does not exist"
Jul 24 12:48:07.456919 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.456907    2566 scope.go:117] "RemoveContainer" containerID="fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c"
Jul 24 12:48:07.466083 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.466062    2566 scope.go:117] "RemoveContainer" containerID="fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c"
Jul 24 12:48:07.466395 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:48:07.466369    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c\": container with ID starting with fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c not found: ID does not exist" containerID="fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c"
Jul 24 12:48:07.466569 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.466409    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c"} err="failed to get container status \"fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c\": rpc error: code = NotFound desc = could not find container \"fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c\": container with ID starting with fde999b21763f27cb01db1e0eb52f7caf123399040939b5a4d74404a60c6f64c not found: ID does not exist"
Jul 24 12:48:07.468585 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.468566    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"]
Jul 24 12:48:07.472279 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.472243    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-266bd0de-78779fff96-c4zn4"]
Jul 24 12:48:07.486599 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.486568    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"]
Jul 24 12:48:07.498996 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.498970    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-266bd0de-84449447bb-kqbm7"]
Jul 24 12:48:07.517459 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.517426    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"]
Jul 24 12:48:07.522798 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:07.522776    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-266bd0de-79b9b79b9b-8296c"]
Jul 24 12:48:08.975656 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:08.975621    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c817bf-546e-4446-abd4-2db4c51c0e5b" path="/var/lib/kubelet/pods/29c817bf-546e-4446-abd4-2db4c51c0e5b/volumes"
Jul 24 12:48:08.976076 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:08.976048    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae7b583c-1c06-497b-8763-af16feb44d1e" path="/var/lib/kubelet/pods/ae7b583c-1c06-497b-8763-af16feb44d1e/volumes"
Jul 24 12:48:08.976368 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:08.976354    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd88b50a-4759-414c-88c1-8a159dc89560" path="/var/lib/kubelet/pods/dd88b50a-4759-414c-88c1-8a159dc89560/volumes"
Jul 24 12:48:11.255267 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:11.255237    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:11.431648 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:11.431617    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:11.670536 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:11.670446    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:12.162181 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:12.162142    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:12.267697 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:12.267663    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:12.369654 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:12.369620    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:13.837067 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:13.837029    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:21.452558 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:21.452523    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:21.566163 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:21.566126    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:21.713602 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:21.713505    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:21.811173 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:21.811142    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:22.183938 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:22.183900    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:25.321340 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:25.321295    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:47.044565 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:47.044537    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:48:47.049222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:47.049196    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:48:48.188071 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:48.188033    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:54.553698 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:54.553661    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:54.654713 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:54.654681    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:48:54.755385 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:48:54.755334    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:10.699160 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:10.699114    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:16.473829 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:16.473769    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:16.545306 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:16.545268    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:16.633309 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:16.633274    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:16.736142 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:16.736053    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:27.253069 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:27.253002    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:28.113291 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:28.113260    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:34.483903 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:34.483869    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:34.593466 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:34.593430    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:34.699574 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:34.699534    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:34.854969 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:34.854931    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:35.667462 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:35.667422    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:35.769986 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:35.769947    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:35.870685 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:35.870655    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:37.582434 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:37.582403    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:43.521288 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:43.521251    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:43.591010 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:43.590976    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:44.240354 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:44.240318    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:50.299510 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:50.299461    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:50.357375 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:50.357346    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:50.568157 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:50.568072    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:50.940825 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:50.940742    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:51.061486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:51.061455    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:54.173933 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.173884    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:54.272312 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.272279    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"]
Jul 24 12:49:54.272663 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.272638    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c817bf-546e-4446-abd4-2db4c51c0e5b" containerName="payload-pre-processing"
Jul 24 12:49:54.272663 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.272653    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c817bf-546e-4446-abd4-2db4c51c0e5b" containerName="payload-pre-processing"
Jul 24 12:49:54.272663 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.272668    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd88b50a-4759-414c-88c1-8a159dc89560" containerName="payload-processing"
Jul 24 12:49:54.272900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.272674    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd88b50a-4759-414c-88c1-8a159dc89560" containerName="payload-processing"
Jul 24 12:49:54.272900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.272689    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae7b583c-1c06-497b-8763-af16feb44d1e" containerName="maas-api"
Jul 24 12:49:54.272900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.272695    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae7b583c-1c06-497b-8763-af16feb44d1e" containerName="maas-api"
Jul 24 12:49:54.272900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.272747    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae7b583c-1c06-497b-8763-af16feb44d1e" containerName="maas-api"
Jul 24 12:49:54.272900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.272753    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c817bf-546e-4446-abd4-2db4c51c0e5b" containerName="payload-pre-processing"
Jul 24 12:49:54.272900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.272761    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd88b50a-4759-414c-88c1-8a159dc89560" containerName="payload-processing"
Jul 24 12:49:54.274482 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.274458    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:54.275896 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.275879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:49:54.280861 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.280844    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e38b46e9-serving-cert\""
Jul 24 12:49:54.287730 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.287710    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"]
Jul 24 12:49:54.351690 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.351663    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:54.430689 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.430609    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v9m9\" (UniqueName: \"kubernetes.io/projected/fbb01297-0560-4977-a58a-dbcd83cf59ef-kube-api-access-9v9m9\") pod \"maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw\" (UID: \"fbb01297-0560-4977-a58a-dbcd83cf59ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:49:54.430689 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.430674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb01297-0560-4977-a58a-dbcd83cf59ef-maas-api-tls\") pod \"maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw\" (UID: \"fbb01297-0560-4977-a58a-dbcd83cf59ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:49:54.430854 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.430725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb01297-0560-4977-a58a-dbcd83cf59ef-openshift-service-ca\") pod \"maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw\" (UID: \"fbb01297-0560-4977-a58a-dbcd83cf59ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:49:54.531508 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.531481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb01297-0560-4977-a58a-dbcd83cf59ef-maas-api-tls\") pod \"maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw\" (UID: \"fbb01297-0560-4977-a58a-dbcd83cf59ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:49:54.531687 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.531514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb01297-0560-4977-a58a-dbcd83cf59ef-openshift-service-ca\") pod \"maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw\" (UID: \"fbb01297-0560-4977-a58a-dbcd83cf59ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:49:54.531687 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.531564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v9m9\" (UniqueName: \"kubernetes.io/projected/fbb01297-0560-4977-a58a-dbcd83cf59ef-kube-api-access-9v9m9\") pod \"maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw\" (UID: \"fbb01297-0560-4977-a58a-dbcd83cf59ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:49:54.532261 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.532235    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb01297-0560-4977-a58a-dbcd83cf59ef-openshift-service-ca\") pod \"maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw\" (UID: \"fbb01297-0560-4977-a58a-dbcd83cf59ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:49:54.533946 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.533925    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb01297-0560-4977-a58a-dbcd83cf59ef-maas-api-tls\") pod \"maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw\" (UID: \"fbb01297-0560-4977-a58a-dbcd83cf59ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:49:54.543997 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.543978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v9m9\" (UniqueName: \"kubernetes.io/projected/fbb01297-0560-4977-a58a-dbcd83cf59ef-kube-api-access-9v9m9\") pod \"maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw\" (UID: \"fbb01297-0560-4977-a58a-dbcd83cf59ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:49:54.550892 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.550856    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:54.586662 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.586644    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:49:54.726933 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.726911    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"]
Jul 24 12:49:54.728820 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:49:54.728785    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb01297_0560_4977_a58a_dbcd83cf59ef.slice/crio-bc69bb07ea7f13f5005f2154fece369b6cdf3e8738cb4e484b8d4dd5f3d921cc WatchSource:0}: Error finding container bc69bb07ea7f13f5005f2154fece369b6cdf3e8738cb4e484b8d4dd5f3d921cc: Status 404 returned error can't find the container with id bc69bb07ea7f13f5005f2154fece369b6cdf3e8738cb4e484b8d4dd5f3d921cc
Jul 24 12:49:54.947665 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.947636    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:54.955609 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:54.955578    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw" event={"ID":"fbb01297-0560-4977-a58a-dbcd83cf59ef","Type":"ContainerStarted","Data":"bc69bb07ea7f13f5005f2154fece369b6cdf3e8738cb4e484b8d4dd5f3d921cc"}
Jul 24 12:49:55.049431 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.049402    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:55.059135 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.059108    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"]
Jul 24 12:49:55.063973 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.063955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"
Jul 24 12:49:55.068138 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.068117    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e38b46e9\""
Jul 24 12:49:55.068138 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.068129    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e38b46e9-dockercfg-477fx\""
Jul 24 12:49:55.073345 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.073321    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"]
Jul 24 12:49:55.085251 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.085232    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"]
Jul 24 12:49:55.088726 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.088708    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"
Jul 24 12:49:55.102732 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.102708    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"]
Jul 24 12:49:55.136261 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.136233    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76055915-0b7a-419c-9db7-3e899eb7bbeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk\" (UID: \"76055915-0b7a-419c-9db7-3e899eb7bbeb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"
Jul 24 12:49:55.136403 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.136284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2dv7\" (UniqueName: \"kubernetes.io/projected/76055915-0b7a-419c-9db7-3e899eb7bbeb-kube-api-access-b2dv7\") pod \"payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk\" (UID: \"76055915-0b7a-419c-9db7-3e899eb7bbeb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"
Jul 24 12:49:55.236743 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.236719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2dv7\" (UniqueName: \"kubernetes.io/projected/76055915-0b7a-419c-9db7-3e899eb7bbeb-kube-api-access-b2dv7\") pod \"payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk\" (UID: \"76055915-0b7a-419c-9db7-3e899eb7bbeb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"
Jul 24 12:49:55.237117 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.236754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c94b7b-1562-4389-a8ad-97b222a785ae-plugins-config-volume\") pod \"payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf\" (UID: \"97c94b7b-1562-4389-a8ad-97b222a785ae\") " pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"
Jul 24 12:49:55.237117 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.236877    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5lp\" (UniqueName: \"kubernetes.io/projected/97c94b7b-1562-4389-a8ad-97b222a785ae-kube-api-access-4j5lp\") pod \"payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf\" (UID: \"97c94b7b-1562-4389-a8ad-97b222a785ae\") " pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"
Jul 24 12:49:55.237218 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.237111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76055915-0b7a-419c-9db7-3e899eb7bbeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk\" (UID: \"76055915-0b7a-419c-9db7-3e899eb7bbeb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"
Jul 24 12:49:55.237699 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.237680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76055915-0b7a-419c-9db7-3e899eb7bbeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk\" (UID: \"76055915-0b7a-419c-9db7-3e899eb7bbeb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"
Jul 24 12:49:55.244882 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.244858    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2dv7\" (UniqueName: \"kubernetes.io/projected/76055915-0b7a-419c-9db7-3e899eb7bbeb-kube-api-access-b2dv7\") pod \"payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk\" (UID: \"76055915-0b7a-419c-9db7-3e899eb7bbeb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"
Jul 24 12:49:55.338255 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.338186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c94b7b-1562-4389-a8ad-97b222a785ae-plugins-config-volume\") pod \"payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf\" (UID: \"97c94b7b-1562-4389-a8ad-97b222a785ae\") " pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"
Jul 24 12:49:55.338255 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.338232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5lp\" (UniqueName: \"kubernetes.io/projected/97c94b7b-1562-4389-a8ad-97b222a785ae-kube-api-access-4j5lp\") pod \"payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf\" (UID: \"97c94b7b-1562-4389-a8ad-97b222a785ae\") " pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"
Jul 24 12:49:55.338822 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.338803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c94b7b-1562-4389-a8ad-97b222a785ae-plugins-config-volume\") pod \"payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf\" (UID: \"97c94b7b-1562-4389-a8ad-97b222a785ae\") " pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"
Jul 24 12:49:55.350383 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.350361    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5lp\" (UniqueName: \"kubernetes.io/projected/97c94b7b-1562-4389-a8ad-97b222a785ae-kube-api-access-4j5lp\") pod \"payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf\" (UID: \"97c94b7b-1562-4389-a8ad-97b222a785ae\") " pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"
Jul 24 12:49:55.376364 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.376339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"
Jul 24 12:49:55.398544 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.398521    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"
Jul 24 12:49:55.528009 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.524077    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"]
Jul 24 12:49:55.528298 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:49:55.528073    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76055915_0b7a_419c_9db7_3e899eb7bbeb.slice/crio-476e0fc17b58f02f15eea247b73d741cff2c8fe7d85b9be94218bbb1126696e4 WatchSource:0}: Error finding container 476e0fc17b58f02f15eea247b73d741cff2c8fe7d85b9be94218bbb1126696e4: Status 404 returned error can't find the container with id 476e0fc17b58f02f15eea247b73d741cff2c8fe7d85b9be94218bbb1126696e4
Jul 24 12:49:55.762269 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.762235    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"]
Jul 24 12:49:55.962747 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.962648    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw" event={"ID":"fbb01297-0560-4977-a58a-dbcd83cf59ef","Type":"ContainerStarted","Data":"c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631"}
Jul 24 12:49:55.962906 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.962770    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:49:55.964524 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.964496    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk" event={"ID":"76055915-0b7a-419c-9db7-3e899eb7bbeb","Type":"ContainerStarted","Data":"c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85"}
Jul 24 12:49:55.964665 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.964545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk" event={"ID":"76055915-0b7a-419c-9db7-3e899eb7bbeb","Type":"ContainerStarted","Data":"476e0fc17b58f02f15eea247b73d741cff2c8fe7d85b9be94218bbb1126696e4"}
Jul 24 12:49:55.964665 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.964644    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"
Jul 24 12:49:55.966401 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.966378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf" event={"ID":"97c94b7b-1562-4389-a8ad-97b222a785ae","Type":"ContainerStarted","Data":"173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c"}
Jul 24 12:49:55.966497 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.966421    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf" event={"ID":"97c94b7b-1562-4389-a8ad-97b222a785ae","Type":"ContainerStarted","Data":"a72c6ea7a0e5d0a8fb5741e818aea96e32685ebbe24a44f7f7b6ce80495c4cc8"}
Jul 24 12:49:55.966497 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.966450    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"
Jul 24 12:49:55.982367 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:55.982327    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw" podStartSLOduration=1.560192463 podStartE2EDuration="1.982303308s" totalImagesPullingTime="422.110845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:49:55.981494542 +0000 UTC m=+1869.509241608" watchObservedRunningTime="2026-07-24 12:49:55.982303308 +0000 UTC m=+1869.510050360"
Jul 24 12:49:56.004956 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:56.004913    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk" podStartSLOduration=1.004900484 podStartE2EDuration="1.004900484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:49:56.002785679 +0000 UTC m=+1869.530532744" watchObservedRunningTime="2026-07-24 12:49:56.004900484 +0000 UTC m=+1869.532647536"
Jul 24 12:49:56.022490 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:56.022451    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf" podStartSLOduration=1.022440528 podStartE2EDuration="1.022440528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:49:56.020058983 +0000 UTC m=+1869.547806035" watchObservedRunningTime="2026-07-24 12:49:56.022440528 +0000 UTC m=+1869.550187608"
Jul 24 12:49:58.132050 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:58.131993    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:49:59.359448 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:49:59.359417    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:00.855321 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:00.855269    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:00.952060 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:00.951993    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:01.547001 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:01.546969    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:01.978682 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:01.978607    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:50:06.975741 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:06.975712    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"
Jul 24 12:50:06.976244 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:06.975774    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"
Jul 24 12:50:07.550544 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:07.550505    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:07.650554 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:07.650519    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:18.537008 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.536958    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"]
Jul 24 12:50:18.537486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.537235    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw" podUID="fbb01297-0560-4977-a58a-dbcd83cf59ef" containerName="maas-api" containerID="cri-o://c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631" gracePeriod=30
Jul 24 12:50:18.565383 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.565358    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:18.591847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.591814    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"]
Jul 24 12:50:18.592080 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.592055    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf" podUID="97c94b7b-1562-4389-a8ad-97b222a785ae" containerName="payload-processing" containerID="cri-o://173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c" gracePeriod=30
Jul 24 12:50:18.600900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.600862    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"]
Jul 24 12:50:18.601144 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.601087    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk" podUID="76055915-0b7a-419c-9db7-3e899eb7bbeb" containerName="payload-pre-processing" containerID="cri-o://c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85" gracePeriod=30
Jul 24 12:50:18.630994 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.630966    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:18.746994 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.746953    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:18.877591 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.877566    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"
Jul 24 12:50:18.894744 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.894713    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"
Jul 24 12:50:18.907356 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.907332    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:50:18.942761 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.942738    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76055915-0b7a-419c-9db7-3e899eb7bbeb-plugins-config-volume\") pod \"76055915-0b7a-419c-9db7-3e899eb7bbeb\" (UID: \"76055915-0b7a-419c-9db7-3e899eb7bbeb\") "
Jul 24 12:50:18.942894 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.942787    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2dv7\" (UniqueName: \"kubernetes.io/projected/76055915-0b7a-419c-9db7-3e899eb7bbeb-kube-api-access-b2dv7\") pod \"76055915-0b7a-419c-9db7-3e899eb7bbeb\" (UID: \"76055915-0b7a-419c-9db7-3e899eb7bbeb\") "
Jul 24 12:50:18.942894 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.942815    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v9m9\" (UniqueName: \"kubernetes.io/projected/fbb01297-0560-4977-a58a-dbcd83cf59ef-kube-api-access-9v9m9\") pod \"fbb01297-0560-4977-a58a-dbcd83cf59ef\" (UID: \"fbb01297-0560-4977-a58a-dbcd83cf59ef\") "
Jul 24 12:50:18.942894 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.942875    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4j5lp\" (UniqueName: \"kubernetes.io/projected/97c94b7b-1562-4389-a8ad-97b222a785ae-kube-api-access-4j5lp\") pod \"97c94b7b-1562-4389-a8ad-97b222a785ae\" (UID: \"97c94b7b-1562-4389-a8ad-97b222a785ae\") "
Jul 24 12:50:18.943084 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.942914    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb01297-0560-4977-a58a-dbcd83cf59ef-openshift-service-ca\") pod \"fbb01297-0560-4977-a58a-dbcd83cf59ef\" (UID: \"fbb01297-0560-4977-a58a-dbcd83cf59ef\") "
Jul 24 12:50:18.943084 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.942979    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb01297-0560-4977-a58a-dbcd83cf59ef-maas-api-tls\") pod \"fbb01297-0560-4977-a58a-dbcd83cf59ef\" (UID: \"fbb01297-0560-4977-a58a-dbcd83cf59ef\") "
Jul 24 12:50:18.943084 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.943006    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c94b7b-1562-4389-a8ad-97b222a785ae-plugins-config-volume\") pod \"97c94b7b-1562-4389-a8ad-97b222a785ae\" (UID: \"97c94b7b-1562-4389-a8ad-97b222a785ae\") "
Jul 24 12:50:18.943222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.943196    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76055915-0b7a-419c-9db7-3e899eb7bbeb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76055915-0b7a-419c-9db7-3e899eb7bbeb" (UID: "76055915-0b7a-419c-9db7-3e899eb7bbeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:50:18.943474 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.943443    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76055915-0b7a-419c-9db7-3e899eb7bbeb-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:50:18.943680 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.943655    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb01297-0560-4977-a58a-dbcd83cf59ef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fbb01297-0560-4977-a58a-dbcd83cf59ef" (UID: "fbb01297-0560-4977-a58a-dbcd83cf59ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:50:18.943768 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.943740    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c94b7b-1562-4389-a8ad-97b222a785ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "97c94b7b-1562-4389-a8ad-97b222a785ae" (UID: "97c94b7b-1562-4389-a8ad-97b222a785ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:50:18.945105 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.945081    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76055915-0b7a-419c-9db7-3e899eb7bbeb-kube-api-access-b2dv7" (OuterVolumeSpecName: "kube-api-access-b2dv7") pod "76055915-0b7a-419c-9db7-3e899eb7bbeb" (UID: "76055915-0b7a-419c-9db7-3e899eb7bbeb"). InnerVolumeSpecName "kube-api-access-b2dv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:50:18.945105 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.945089    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb01297-0560-4977-a58a-dbcd83cf59ef-kube-api-access-9v9m9" (OuterVolumeSpecName: "kube-api-access-9v9m9") pod "fbb01297-0560-4977-a58a-dbcd83cf59ef" (UID: "fbb01297-0560-4977-a58a-dbcd83cf59ef"). InnerVolumeSpecName "kube-api-access-9v9m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:50:18.945692 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.945674    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c94b7b-1562-4389-a8ad-97b222a785ae-kube-api-access-4j5lp" (OuterVolumeSpecName: "kube-api-access-4j5lp") pod "97c94b7b-1562-4389-a8ad-97b222a785ae" (UID: "97c94b7b-1562-4389-a8ad-97b222a785ae"). InnerVolumeSpecName "kube-api-access-4j5lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:50:18.945766 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:18.945743    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb01297-0560-4977-a58a-dbcd83cf59ef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fbb01297-0560-4977-a58a-dbcd83cf59ef" (UID: "fbb01297-0560-4977-a58a-dbcd83cf59ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:50:19.043916 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.043891    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j5lp\" (UniqueName: \"kubernetes.io/projected/97c94b7b-1562-4389-a8ad-97b222a785ae-kube-api-access-4j5lp\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:50:19.043916 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.043916    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb01297-0560-4977-a58a-dbcd83cf59ef-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:50:19.044066 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.043926    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb01297-0560-4977-a58a-dbcd83cf59ef-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:50:19.044066 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.043934    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c94b7b-1562-4389-a8ad-97b222a785ae-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:50:19.044066 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.043943    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2dv7\" (UniqueName: \"kubernetes.io/projected/76055915-0b7a-419c-9db7-3e899eb7bbeb-kube-api-access-b2dv7\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:50:19.044066 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.043950    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v9m9\" (UniqueName: \"kubernetes.io/projected/fbb01297-0560-4977-a58a-dbcd83cf59ef-kube-api-access-9v9m9\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:50:19.073920 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.073853    2566 generic.go:356] "Generic (PLEG): container finished" podID="97c94b7b-1562-4389-a8ad-97b222a785ae" containerID="173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c" exitCode=0
Jul 24 12:50:19.073920 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.073905    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"
Jul 24 12:50:19.074104 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.073941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf" event={"ID":"97c94b7b-1562-4389-a8ad-97b222a785ae","Type":"ContainerDied","Data":"173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c"}
Jul 24 12:50:19.074104 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.073994    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf" event={"ID":"97c94b7b-1562-4389-a8ad-97b222a785ae","Type":"ContainerDied","Data":"a72c6ea7a0e5d0a8fb5741e818aea96e32685ebbe24a44f7f7b6ce80495c4cc8"}
Jul 24 12:50:19.074104 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.074026    2566 scope.go:117] "RemoveContainer" containerID="173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c"
Jul 24 12:50:19.075439 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.075418    2566 generic.go:356] "Generic (PLEG): container finished" podID="fbb01297-0560-4977-a58a-dbcd83cf59ef" containerID="c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631" exitCode=0
Jul 24 12:50:19.075534 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.075450    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw" event={"ID":"fbb01297-0560-4977-a58a-dbcd83cf59ef","Type":"ContainerDied","Data":"c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631"}
Jul 24 12:50:19.075595 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.075570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw" event={"ID":"fbb01297-0560-4977-a58a-dbcd83cf59ef","Type":"ContainerDied","Data":"bc69bb07ea7f13f5005f2154fece369b6cdf3e8738cb4e484b8d4dd5f3d921cc"}
Jul 24 12:50:19.075595 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.075458    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"
Jul 24 12:50:19.077108 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.077082    2566 generic.go:356] "Generic (PLEG): container finished" podID="76055915-0b7a-419c-9db7-3e899eb7bbeb" containerID="c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85" exitCode=0
Jul 24 12:50:19.077204 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.077128    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"
Jul 24 12:50:19.077204 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.077153    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk" event={"ID":"76055915-0b7a-419c-9db7-3e899eb7bbeb","Type":"ContainerDied","Data":"c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85"}
Jul 24 12:50:19.077204 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.077192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk" event={"ID":"76055915-0b7a-419c-9db7-3e899eb7bbeb","Type":"ContainerDied","Data":"476e0fc17b58f02f15eea247b73d741cff2c8fe7d85b9be94218bbb1126696e4"}
Jul 24 12:50:19.083612 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.083598    2566 scope.go:117] "RemoveContainer" containerID="173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c"
Jul 24 12:50:19.083881 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:50:19.083853    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c\": container with ID starting with 173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c not found: ID does not exist" containerID="173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c"
Jul 24 12:50:19.083938 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.083879    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c"} err="failed to get container status \"173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c\": rpc error: code = NotFound desc = could not find container \"173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c\": container with ID starting with 173c5e94d6a383dbd4f0d7747d56ec97047fc1b160b597d634afb9731f95e94c not found: ID does not exist"
Jul 24 12:50:19.083938 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.083895    2566 scope.go:117] "RemoveContainer" containerID="c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631"
Jul 24 12:50:19.091375 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.091358    2566 scope.go:117] "RemoveContainer" containerID="c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631"
Jul 24 12:50:19.091603 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:50:19.091587    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631\": container with ID starting with c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631 not found: ID does not exist" containerID="c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631"
Jul 24 12:50:19.091648 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.091610    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631"} err="failed to get container status \"c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631\": rpc error: code = NotFound desc = could not find container \"c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631\": container with ID starting with c6b2f83ae52dd92cffc48e28e52ae9c9897cbd81644f8501f1fd65975132c631 not found: ID does not exist"
Jul 24 12:50:19.091648 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.091625    2566 scope.go:117] "RemoveContainer" containerID="c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85"
Jul 24 12:50:19.098854 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.098837    2566 scope.go:117] "RemoveContainer" containerID="c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85"
Jul 24 12:50:19.099126 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:50:19.099082    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85\": container with ID starting with c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85 not found: ID does not exist" containerID="c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85"
Jul 24 12:50:19.099231 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.099128    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85"} err="failed to get container status \"c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85\": rpc error: code = NotFound desc = could not find container \"c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85\": container with ID starting with c7c46dd19ec15dbb480e2d3db7b7f820f700f4d74f023b37b1053f1e5d536a85 not found: ID does not exist"
Jul 24 12:50:19.100803 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.100788    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"]
Jul 24 12:50:19.105147 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.105127    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e38b46e9-95c654dc6-hwbgk"]
Jul 24 12:50:19.117436 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.117417    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"]
Jul 24 12:50:19.119047 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.119012    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e38b46e9-6f6455694b-xwmwf"]
Jul 24 12:50:19.131653 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.131630    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"]
Jul 24 12:50:19.135626 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:19.135607    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e38b46e9-6864cdf7c9-wrndw"]
Jul 24 12:50:20.975355 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:20.975323    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76055915-0b7a-419c-9db7-3e899eb7bbeb" path="/var/lib/kubelet/pods/76055915-0b7a-419c-9db7-3e899eb7bbeb/volumes"
Jul 24 12:50:20.975773 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:20.975758    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c94b7b-1562-4389-a8ad-97b222a785ae" path="/var/lib/kubelet/pods/97c94b7b-1562-4389-a8ad-97b222a785ae/volumes"
Jul 24 12:50:20.976124 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:20.976111    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbb01297-0560-4977-a58a-dbcd83cf59ef" path="/var/lib/kubelet/pods/fbb01297-0560-4977-a58a-dbcd83cf59ef/volumes"
Jul 24 12:50:23.184007 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:23.183968    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:23.586380 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:23.586337    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:29.398363 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:29.398324    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:29.498294 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:29.498259    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:29.669773 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:29.669674    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:29.918423 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:29.918388    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:30.041833 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:30.041803    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:30.139824 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:30.139791    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:30.276817 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:30.276786    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:36.264297 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:36.264264    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:36.374855 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:36.374819    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:36.466660 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:36.466622    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:36.645638 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:36.645540    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:42.470633 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:42.470572    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:42.574775 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:42.574747    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:49.322915 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:49.322877    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:49.654446 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:49.654357    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:49.791908 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:49.791876    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:52.907879 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.907847    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:52.990688 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.990659    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"]
Jul 24 12:50:52.991341 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.991308    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c94b7b-1562-4389-a8ad-97b222a785ae" containerName="payload-processing"
Jul 24 12:50:52.991496 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.991484    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c94b7b-1562-4389-a8ad-97b222a785ae" containerName="payload-processing"
Jul 24 12:50:52.991587 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.991577    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76055915-0b7a-419c-9db7-3e899eb7bbeb" containerName="payload-pre-processing"
Jul 24 12:50:52.991666 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.991657    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76055915-0b7a-419c-9db7-3e899eb7bbeb" containerName="payload-pre-processing"
Jul 24 12:50:52.991780 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.991763    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb01297-0560-4977-a58a-dbcd83cf59ef" containerName="maas-api"
Jul 24 12:50:52.991780 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.991779    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb01297-0560-4977-a58a-dbcd83cf59ef" containerName="maas-api"
Jul 24 12:50:52.991914 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.991894    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbb01297-0560-4977-a58a-dbcd83cf59ef" containerName="maas-api"
Jul 24 12:50:52.991914 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.991905    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="76055915-0b7a-419c-9db7-3e899eb7bbeb" containerName="payload-pre-processing"
Jul 24 12:50:52.991981 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.991918    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c94b7b-1562-4389-a8ad-97b222a785ae" containerName="payload-processing"
Jul 24 12:50:52.995431 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.995412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:50:52.998306 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:52.998285    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f55af331-serving-cert\""
Jul 24 12:50:53.006216 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.006192    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"]
Jul 24 12:50:53.006403 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.006225    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:53.084254 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.084192    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:53.145947 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.145909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-maas-api-tls\") pod \"maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw\" (UID: \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:50:53.146169 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.145999    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-openshift-service-ca\") pod \"maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw\" (UID: \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:50:53.146169 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.146081    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh25l\" (UniqueName: \"kubernetes.io/projected/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-kube-api-access-wh25l\") pod \"maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw\" (UID: \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:50:53.228720 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.228632    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:53.247526 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.247498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh25l\" (UniqueName: \"kubernetes.io/projected/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-kube-api-access-wh25l\") pod \"maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw\" (UID: \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:50:53.247661 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.247560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-maas-api-tls\") pod \"maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw\" (UID: \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:50:53.247661 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.247616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-openshift-service-ca\") pod \"maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw\" (UID: \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:50:53.248357 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.248309    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-openshift-service-ca\") pod \"maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw\" (UID: \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:50:53.250084 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.250065    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-maas-api-tls\") pod \"maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw\" (UID: \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:50:53.259533 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.259508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh25l\" (UniqueName: \"kubernetes.io/projected/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-kube-api-access-wh25l\") pod \"maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw\" (UID: \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:50:53.307271 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.307243    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:50:53.464741 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.464715    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"]
Jul 24 12:50:53.466669 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:50:53.466639    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2c55c1_7a8f_4e4f_a27b_8e4b180fa3c4.slice/crio-bc9cbaf42dcd3bd392eb0949e6622968a3d9edc24c2d7636e5ef927acfad6a3f WatchSource:0}: Error finding container bc9cbaf42dcd3bd392eb0949e6622968a3d9edc24c2d7636e5ef927acfad6a3f: Status 404 returned error can't find the container with id bc9cbaf42dcd3bd392eb0949e6622968a3d9edc24c2d7636e5ef927acfad6a3f
Jul 24 12:50:53.468135 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.468115    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 12:50:53.745775 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.745741    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:53.842948 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.842916    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"]
Jul 24 12:50:53.846474 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.846442    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"
Jul 24 12:50:53.847729 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.847703    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:53.849919 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.849895    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f55af331-dockercfg-x2mbg\""
Jul 24 12:50:53.850037 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.849941    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f55af331\""
Jul 24 12:50:53.857864 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.857842    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"]
Jul 24 12:50:53.869040 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.867755    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"]
Jul 24 12:50:53.873055 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.873032    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"
Jul 24 12:50:53.883248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.883220    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"]
Jul 24 12:50:53.955135 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.955109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6ngv\" (UniqueName: \"kubernetes.io/projected/d32c90dc-3403-44ca-9cfb-be41858888e1-kube-api-access-j6ngv\") pod \"payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx\" (UID: \"d32c90dc-3403-44ca-9cfb-be41858888e1\") " pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"
Jul 24 12:50:53.955504 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.955141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1238f62d-c42a-4618-bde4-3cece15b4a93-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk\" (UID: \"1238f62d-c42a-4618-bde4-3cece15b4a93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"
Jul 24 12:50:53.955504 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.955159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45jbx\" (UniqueName: \"kubernetes.io/projected/1238f62d-c42a-4618-bde4-3cece15b4a93-kube-api-access-45jbx\") pod \"payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk\" (UID: \"1238f62d-c42a-4618-bde4-3cece15b4a93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"
Jul 24 12:50:53.955504 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:53.955279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32c90dc-3403-44ca-9cfb-be41858888e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx\" (UID: \"d32c90dc-3403-44ca-9cfb-be41858888e1\") " pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"
Jul 24 12:50:54.056129 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.056046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32c90dc-3403-44ca-9cfb-be41858888e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx\" (UID: \"d32c90dc-3403-44ca-9cfb-be41858888e1\") " pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"
Jul 24 12:50:54.056129 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.056130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6ngv\" (UniqueName: \"kubernetes.io/projected/d32c90dc-3403-44ca-9cfb-be41858888e1-kube-api-access-j6ngv\") pod \"payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx\" (UID: \"d32c90dc-3403-44ca-9cfb-be41858888e1\") " pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"
Jul 24 12:50:54.056325 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.056146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1238f62d-c42a-4618-bde4-3cece15b4a93-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk\" (UID: \"1238f62d-c42a-4618-bde4-3cece15b4a93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"
Jul 24 12:50:54.056325 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.056162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45jbx\" (UniqueName: \"kubernetes.io/projected/1238f62d-c42a-4618-bde4-3cece15b4a93-kube-api-access-45jbx\") pod \"payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk\" (UID: \"1238f62d-c42a-4618-bde4-3cece15b4a93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"
Jul 24 12:50:54.056921 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.056898    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32c90dc-3403-44ca-9cfb-be41858888e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx\" (UID: \"d32c90dc-3403-44ca-9cfb-be41858888e1\") " pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"
Jul 24 12:50:54.056960 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.056898    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1238f62d-c42a-4618-bde4-3cece15b4a93-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk\" (UID: \"1238f62d-c42a-4618-bde4-3cece15b4a93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"
Jul 24 12:50:54.069648 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.069620    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6ngv\" (UniqueName: \"kubernetes.io/projected/d32c90dc-3403-44ca-9cfb-be41858888e1-kube-api-access-j6ngv\") pod \"payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx\" (UID: \"d32c90dc-3403-44ca-9cfb-be41858888e1\") " pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"
Jul 24 12:50:54.069855 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.069835    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45jbx\" (UniqueName: \"kubernetes.io/projected/1238f62d-c42a-4618-bde4-3cece15b4a93-kube-api-access-45jbx\") pod \"payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk\" (UID: \"1238f62d-c42a-4618-bde4-3cece15b4a93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"
Jul 24 12:50:54.158760 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.158729    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"
Jul 24 12:50:54.185885 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.185859    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"
Jul 24 12:50:54.246760 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.246590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw" event={"ID":"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4","Type":"ContainerStarted","Data":"d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e"}
Jul 24 12:50:54.246760 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.246655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw" event={"ID":"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4","Type":"ContainerStarted","Data":"bc9cbaf42dcd3bd392eb0949e6622968a3d9edc24c2d7636e5ef927acfad6a3f"}
Jul 24 12:50:54.247139 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.247072    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:50:54.275918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.275733    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw" podStartSLOduration=1.884512746 podStartE2EDuration="2.27571036s" totalImagesPullingTime="391.197614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:50:54.274641765 +0000 UTC m=+1927.802388819" watchObservedRunningTime="2026-07-24 12:50:54.27571036 +0000 UTC m=+1927.803457414"
Jul 24 12:50:54.326107 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.326012    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"]
Jul 24 12:50:54.327652 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:50:54.327607    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1238f62d_c42a_4618_bde4_3cece15b4a93.slice/crio-53ccb3d07f07266bd6994141764c91709940d7104a3f907fb6290db633b5cf2e WatchSource:0}: Error finding container 53ccb3d07f07266bd6994141764c91709940d7104a3f907fb6290db633b5cf2e: Status 404 returned error can't find the container with id 53ccb3d07f07266bd6994141764c91709940d7104a3f907fb6290db633b5cf2e
Jul 24 12:50:54.353117 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.353090    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"]
Jul 24 12:50:54.353910 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:50:54.353889    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32c90dc_3403_44ca_9cfb_be41858888e1.slice/crio-70cd94b03e71b0d045ab94489ec536f54a60e7f63f31c827d2b385106c103677 WatchSource:0}: Error finding container 70cd94b03e71b0d045ab94489ec536f54a60e7f63f31c827d2b385106c103677: Status 404 returned error can't find the container with id 70cd94b03e71b0d045ab94489ec536f54a60e7f63f31c827d2b385106c103677
Jul 24 12:50:54.853239 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.853203    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:54.953437 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:54.953405    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:50:55.252862 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:55.252815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk" event={"ID":"1238f62d-c42a-4618-bde4-3cece15b4a93","Type":"ContainerStarted","Data":"4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d"}
Jul 24 12:50:55.253327 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:55.252889    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk" event={"ID":"1238f62d-c42a-4618-bde4-3cece15b4a93","Type":"ContainerStarted","Data":"53ccb3d07f07266bd6994141764c91709940d7104a3f907fb6290db633b5cf2e"}
Jul 24 12:50:55.253327 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:55.253001    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"
Jul 24 12:50:55.254526 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:55.254500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx" event={"ID":"d32c90dc-3403-44ca-9cfb-be41858888e1","Type":"ContainerStarted","Data":"829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b"}
Jul 24 12:50:55.254652 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:55.254539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx" event={"ID":"d32c90dc-3403-44ca-9cfb-be41858888e1","Type":"ContainerStarted","Data":"70cd94b03e71b0d045ab94489ec536f54a60e7f63f31c827d2b385106c103677"}
Jul 24 12:50:55.254652 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:55.254594    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"
Jul 24 12:50:55.271310 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:55.271259    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk" podStartSLOduration=2.27124166 podStartE2EDuration="2.27124166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:50:55.269705731 +0000 UTC m=+1928.797452788" watchObservedRunningTime="2026-07-24 12:50:55.27124166 +0000 UTC m=+1928.798988714"
Jul 24 12:50:55.290913 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:55.290871    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx" podStartSLOduration=2.29086268 podStartE2EDuration="2.29086268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:50:55.289380666 +0000 UTC m=+1928.817127720" watchObservedRunningTime="2026-07-24 12:50:55.29086268 +0000 UTC m=+1928.818609732"
Jul 24 12:50:57.446800 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:50:57.446764    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:51:00.259649 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:00.259621    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:51:02.639575 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:02.639542    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:51:06.261976 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:06.261941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"
Jul 24 12:51:06.262385 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:06.262054    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"
Jul 24 12:51:18.090688 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:18.090649    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:51:18.156247 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:18.156218    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:51:42.388752 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.388658    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"]
Jul 24 12:51:42.389340 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.388944    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw" podUID="1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4" containerName="maas-api" containerID="cri-o://d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e" gracePeriod=30
Jul 24 12:51:42.417369 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.417342    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:51:42.439637 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.439611    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"]
Jul 24 12:51:42.439850 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.439826    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx" podUID="d32c90dc-3403-44ca-9cfb-be41858888e1" containerName="payload-processing" containerID="cri-o://829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b" gracePeriod=30
Jul 24 12:51:42.450759 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.450187    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"]
Jul 24 12:51:42.450759 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.450412    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk" podUID="1238f62d-c42a-4618-bde4-3cece15b4a93" containerName="payload-pre-processing" containerID="cri-o://4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d" gracePeriod=30
Jul 24 12:51:42.484209 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.484168    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:51:42.593114 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.589908    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:51:42.735692 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.735664    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"
Jul 24 12:51:42.738462 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.738443    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:51:42.753839 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.753809    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"
Jul 24 12:51:42.887739 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.887709    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1238f62d-c42a-4618-bde4-3cece15b4a93-plugins-config-volume\") pod \"1238f62d-c42a-4618-bde4-3cece15b4a93\" (UID: \"1238f62d-c42a-4618-bde4-3cece15b4a93\") "
Jul 24 12:51:42.887943 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.887756    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh25l\" (UniqueName: \"kubernetes.io/projected/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-kube-api-access-wh25l\") pod \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\" (UID: \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\") "
Jul 24 12:51:42.887943 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.887784    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-openshift-service-ca\") pod \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\" (UID: \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\") "
Jul 24 12:51:42.887943 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.887849    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-maas-api-tls\") pod \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\" (UID: \"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4\") "
Jul 24 12:51:42.887943 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.887876    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32c90dc-3403-44ca-9cfb-be41858888e1-plugins-config-volume\") pod \"d32c90dc-3403-44ca-9cfb-be41858888e1\" (UID: \"d32c90dc-3403-44ca-9cfb-be41858888e1\") "
Jul 24 12:51:42.887943 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.887920    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45jbx\" (UniqueName: \"kubernetes.io/projected/1238f62d-c42a-4618-bde4-3cece15b4a93-kube-api-access-45jbx\") pod \"1238f62d-c42a-4618-bde4-3cece15b4a93\" (UID: \"1238f62d-c42a-4618-bde4-3cece15b4a93\") "
Jul 24 12:51:42.888276 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.887948    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6ngv\" (UniqueName: \"kubernetes.io/projected/d32c90dc-3403-44ca-9cfb-be41858888e1-kube-api-access-j6ngv\") pod \"d32c90dc-3403-44ca-9cfb-be41858888e1\" (UID: \"d32c90dc-3403-44ca-9cfb-be41858888e1\") "
Jul 24 12:51:42.888333 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.888273    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4" (UID: "1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:51:42.888333 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.888293    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1238f62d-c42a-4618-bde4-3cece15b4a93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1238f62d-c42a-4618-bde4-3cece15b4a93" (UID: "1238f62d-c42a-4618-bde4-3cece15b4a93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:51:42.888840 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.888806    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32c90dc-3403-44ca-9cfb-be41858888e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d32c90dc-3403-44ca-9cfb-be41858888e1" (UID: "d32c90dc-3403-44ca-9cfb-be41858888e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:51:42.890094 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.890072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-kube-api-access-wh25l" (OuterVolumeSpecName: "kube-api-access-wh25l") pod "1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4" (UID: "1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4"). InnerVolumeSpecName "kube-api-access-wh25l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:51:42.890256 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.890238    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32c90dc-3403-44ca-9cfb-be41858888e1-kube-api-access-j6ngv" (OuterVolumeSpecName: "kube-api-access-j6ngv") pod "d32c90dc-3403-44ca-9cfb-be41858888e1" (UID: "d32c90dc-3403-44ca-9cfb-be41858888e1"). InnerVolumeSpecName "kube-api-access-j6ngv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:51:42.890638 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.890608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1238f62d-c42a-4618-bde4-3cece15b4a93-kube-api-access-45jbx" (OuterVolumeSpecName: "kube-api-access-45jbx") pod "1238f62d-c42a-4618-bde4-3cece15b4a93" (UID: "1238f62d-c42a-4618-bde4-3cece15b4a93"). InnerVolumeSpecName "kube-api-access-45jbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:51:42.890638 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.890617    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4" (UID: "1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:51:42.989915 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.989687    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1238f62d-c42a-4618-bde4-3cece15b4a93-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:51:42.989915 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.989710    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh25l\" (UniqueName: \"kubernetes.io/projected/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-kube-api-access-wh25l\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:51:42.989915 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.989723    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:51:42.989915 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.989737    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:51:42.989915 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.989750    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32c90dc-3403-44ca-9cfb-be41858888e1-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:51:42.989915 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.989759    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45jbx\" (UniqueName: \"kubernetes.io/projected/1238f62d-c42a-4618-bde4-3cece15b4a93-kube-api-access-45jbx\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:51:42.989915 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:42.989767    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6ngv\" (UniqueName: \"kubernetes.io/projected/d32c90dc-3403-44ca-9cfb-be41858888e1-kube-api-access-j6ngv\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:51:43.504685 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.504651    2566 generic.go:356] "Generic (PLEG): container finished" podID="d32c90dc-3403-44ca-9cfb-be41858888e1" containerID="829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b" exitCode=0
Jul 24 12:51:43.505153 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.504715    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"
Jul 24 12:51:43.505153 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.504726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx" event={"ID":"d32c90dc-3403-44ca-9cfb-be41858888e1","Type":"ContainerDied","Data":"829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b"}
Jul 24 12:51:43.505153 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.504781    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx" event={"ID":"d32c90dc-3403-44ca-9cfb-be41858888e1","Type":"ContainerDied","Data":"70cd94b03e71b0d045ab94489ec536f54a60e7f63f31c827d2b385106c103677"}
Jul 24 12:51:43.505153 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.504804    2566 scope.go:117] "RemoveContainer" containerID="829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b"
Jul 24 12:51:43.506310 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.506283    2566 generic.go:356] "Generic (PLEG): container finished" podID="1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4" containerID="d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e" exitCode=0
Jul 24 12:51:43.506446 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.506342    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"
Jul 24 12:51:43.506446 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.506369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw" event={"ID":"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4","Type":"ContainerDied","Data":"d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e"}
Jul 24 12:51:43.506446 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.506406    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw" event={"ID":"1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4","Type":"ContainerDied","Data":"bc9cbaf42dcd3bd392eb0949e6622968a3d9edc24c2d7636e5ef927acfad6a3f"}
Jul 24 12:51:43.507824 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.507801    2566 generic.go:356] "Generic (PLEG): container finished" podID="1238f62d-c42a-4618-bde4-3cece15b4a93" containerID="4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d" exitCode=0
Jul 24 12:51:43.507915 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.507838    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk" event={"ID":"1238f62d-c42a-4618-bde4-3cece15b4a93","Type":"ContainerDied","Data":"4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d"}
Jul 24 12:51:43.507915 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.507873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk" event={"ID":"1238f62d-c42a-4618-bde4-3cece15b4a93","Type":"ContainerDied","Data":"53ccb3d07f07266bd6994141764c91709940d7104a3f907fb6290db633b5cf2e"}
Jul 24 12:51:43.507915 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.507876    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"
Jul 24 12:51:43.514565 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.514551    2566 scope.go:117] "RemoveContainer" containerID="829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b"
Jul 24 12:51:43.514836 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:51:43.514816    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b\": container with ID starting with 829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b not found: ID does not exist" containerID="829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b"
Jul 24 12:51:43.514909 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.514843    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b"} err="failed to get container status \"829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b\": rpc error: code = NotFound desc = could not find container \"829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b\": container with ID starting with 829412c7e2a78dae84609c8646b1a58b8c255b430906e60c3a03b6bf494cd53b not found: ID does not exist"
Jul 24 12:51:43.514909 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.514860    2566 scope.go:117] "RemoveContainer" containerID="d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e"
Jul 24 12:51:43.523248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.523230    2566 scope.go:117] "RemoveContainer" containerID="d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e"
Jul 24 12:51:43.523498 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:51:43.523480    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e\": container with ID starting with d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e not found: ID does not exist" containerID="d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e"
Jul 24 12:51:43.523547 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.523508    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e"} err="failed to get container status \"d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e\": rpc error: code = NotFound desc = could not find container \"d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e\": container with ID starting with d6e82f702df6d1a9d81d22255c306ee4718d6b844161fc3fd711988bfdff358e not found: ID does not exist"
Jul 24 12:51:43.523547 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.523524    2566 scope.go:117] "RemoveContainer" containerID="4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d"
Jul 24 12:51:43.531254 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.531239    2566 scope.go:117] "RemoveContainer" containerID="4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d"
Jul 24 12:51:43.531510 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:51:43.531491    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d\": container with ID starting with 4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d not found: ID does not exist" containerID="4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d"
Jul 24 12:51:43.531576 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.531520    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d"} err="failed to get container status \"4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d\": rpc error: code = NotFound desc = could not find container \"4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d\": container with ID starting with 4fa106e85248d6c3e53078016c6dc768f835de9c63d0e925f492ed02b39d825d not found: ID does not exist"
Jul 24 12:51:43.532222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.532201    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"]
Jul 24 12:51:43.538132 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.538112    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f55af331-555fc65d48-nfkbk"]
Jul 24 12:51:43.550237 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.550217    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"]
Jul 24 12:51:43.554599 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.554578    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f55af331-9f6cc47bf-9wtvw"]
Jul 24 12:51:43.567239 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.567215    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"]
Jul 24 12:51:43.570623 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:43.570602    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f55af331-64cb444d54-nmjwx"]
Jul 24 12:51:44.975548 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:44.975516    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1238f62d-c42a-4618-bde4-3cece15b4a93" path="/var/lib/kubelet/pods/1238f62d-c42a-4618-bde4-3cece15b4a93/volumes"
Jul 24 12:51:44.975967 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:44.975893    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4" path="/var/lib/kubelet/pods/1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4/volumes"
Jul 24 12:51:44.976251 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:44.976229    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d32c90dc-3403-44ca-9cfb-be41858888e1" path="/var/lib/kubelet/pods/d32c90dc-3403-44ca-9cfb-be41858888e1/volumes"
Jul 24 12:51:46.969486 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:46.969436    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:51:50.448503 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:50.448464    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:51:55.521229 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:51:55.521175    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:32.519753 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:32.519720    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:32.679445 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:32.679387    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:36.545381 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:36.545343    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:38.492030 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:38.491993    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:38.553042 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:38.552997    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:38.781437 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:38.781403    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:39.062053 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:39.061941    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:40.119973 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:40.119939    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:40.218895 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:40.218860    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:40.317668 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:40.317644    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:41.083942 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:41.083914    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:41.173818 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:41.173785    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:41.275798 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:41.275770    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:42.710605 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:42.710573    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:48.809050 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:48.808996    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:49.768692 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:49.768659    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:49.824150 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:49.824120    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:56.012274 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:56.012235    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:56.119104 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:56.119069    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:52:56.214287 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:52:56.214253    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:06.778204 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:06.778162    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:07.140215 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:07.140125    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:28.149244 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:28.149202    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:34.488600 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:34.488555    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:34.589816 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:34.589785    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:34.858471 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:34.858429    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:35.207357 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:35.207269    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:35.429339 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:35.429297    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:37.982957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:37.982925    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:38.052077 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.052046    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"]
Jul 24 12:53:38.052433 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.052421    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1238f62d-c42a-4618-bde4-3cece15b4a93" containerName="payload-pre-processing"
Jul 24 12:53:38.052480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.052434    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1238f62d-c42a-4618-bde4-3cece15b4a93" containerName="payload-pre-processing"
Jul 24 12:53:38.052480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.052447    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32c90dc-3403-44ca-9cfb-be41858888e1" containerName="payload-processing"
Jul 24 12:53:38.052480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.052453    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32c90dc-3403-44ca-9cfb-be41858888e1" containerName="payload-processing"
Jul 24 12:53:38.052480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.052466    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4" containerName="maas-api"
Jul 24 12:53:38.052480 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.052471    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4" containerName="maas-api"
Jul 24 12:53:38.052648 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.052527    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d2c55c1-7a8f-4e4f-a27b-8e4b180fa3c4" containerName="maas-api"
Jul 24 12:53:38.052648 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.052538    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1238f62d-c42a-4618-bde4-3cece15b4a93" containerName="payload-pre-processing"
Jul 24 12:53:38.052648 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.052547    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d32c90dc-3403-44ca-9cfb-be41858888e1" containerName="payload-processing"
Jul 24 12:53:38.055829 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.055810    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:38.059009 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.058992    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-dd4baf-serving-cert\""
Jul 24 12:53:38.070725 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.070704    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"]
Jul 24 12:53:38.073285 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.073262    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:38.147210 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.147176    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5a71b66-9be8-4010-b03c-375719b4d492-maas-api-tls\") pod \"maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr\" (UID: \"e5a71b66-9be8-4010-b03c-375719b4d492\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:38.147376 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.147210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a71b66-9be8-4010-b03c-375719b4d492-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr\" (UID: \"e5a71b66-9be8-4010-b03c-375719b4d492\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:38.147376 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.147238    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pcz8\" (UniqueName: \"kubernetes.io/projected/e5a71b66-9be8-4010-b03c-375719b4d492-kube-api-access-2pcz8\") pod \"maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr\" (UID: \"e5a71b66-9be8-4010-b03c-375719b4d492\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:38.172662 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.172632    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:38.248663 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.248638    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5a71b66-9be8-4010-b03c-375719b4d492-maas-api-tls\") pod \"maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr\" (UID: \"e5a71b66-9be8-4010-b03c-375719b4d492\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:38.248825 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.248669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a71b66-9be8-4010-b03c-375719b4d492-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr\" (UID: \"e5a71b66-9be8-4010-b03c-375719b4d492\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:38.248825 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.248687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pcz8\" (UniqueName: \"kubernetes.io/projected/e5a71b66-9be8-4010-b03c-375719b4d492-kube-api-access-2pcz8\") pod \"maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr\" (UID: \"e5a71b66-9be8-4010-b03c-375719b4d492\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:38.249361 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.249336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a71b66-9be8-4010-b03c-375719b4d492-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr\" (UID: \"e5a71b66-9be8-4010-b03c-375719b4d492\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:38.251111 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.251083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5a71b66-9be8-4010-b03c-375719b4d492-maas-api-tls\") pod \"maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr\" (UID: \"e5a71b66-9be8-4010-b03c-375719b4d492\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:38.258579 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.258554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pcz8\" (UniqueName: \"kubernetes.io/projected/e5a71b66-9be8-4010-b03c-375719b4d492-kube-api-access-2pcz8\") pod \"maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr\" (UID: \"e5a71b66-9be8-4010-b03c-375719b4d492\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:38.274434 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.274408    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:38.367574 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.367554    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:38.689394 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.689307    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:38.721138 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.721111    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"]
Jul 24 12:53:38.723373 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:53:38.723348    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a71b66_9be8_4010_b03c_375719b4d492.slice/crio-2dc8e8c3e612c4d02a62e2e14a56e4629c028057068ab5fd3113cf316fde927f WatchSource:0}: Error finding container 2dc8e8c3e612c4d02a62e2e14a56e4629c028057068ab5fd3113cf316fde927f: Status 404 returned error can't find the container with id 2dc8e8c3e612c4d02a62e2e14a56e4629c028057068ab5fd3113cf316fde927f
Jul 24 12:53:38.791753 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.791717    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:38.819823 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.819798    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"]
Jul 24 12:53:38.824396 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.824381    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"
Jul 24 12:53:38.828145 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.828128    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-dd4baf-dockercfg-66fvh\""
Jul 24 12:53:38.828258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.828127    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-dd4baf\""
Jul 24 12:53:38.837348 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.837328    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"]
Jul 24 12:53:38.874656 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.874629    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"]
Jul 24 12:53:38.877918 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.877903    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"
Jul 24 12:53:38.888778 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.888758    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"]
Jul 24 12:53:38.956435 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.956361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c76152-efac-4a76-bd1c-8d6afa6e105f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb\" (UID: \"a6c76152-efac-4a76-bd1c-8d6afa6e105f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"
Jul 24 12:53:38.956564 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.956431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98xw\" (UniqueName: \"kubernetes.io/projected/a6c76152-efac-4a76-bd1c-8d6afa6e105f-kube-api-access-b98xw\") pod \"payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb\" (UID: \"a6c76152-efac-4a76-bd1c-8d6afa6e105f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"
Jul 24 12:53:38.956564 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.956482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpff\" (UniqueName: \"kubernetes.io/projected/7bc085cb-4182-428a-925b-4c6306b1c2d7-kube-api-access-hwpff\") pod \"payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk\" (UID: \"7bc085cb-4182-428a-925b-4c6306b1c2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"
Jul 24 12:53:38.956564 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:38.956509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc085cb-4182-428a-925b-4c6306b1c2d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk\" (UID: \"7bc085cb-4182-428a-925b-4c6306b1c2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"
Jul 24 12:53:39.046361 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.046324    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr" event={"ID":"e5a71b66-9be8-4010-b03c-375719b4d492","Type":"ContainerStarted","Data":"2dc8e8c3e612c4d02a62e2e14a56e4629c028057068ab5fd3113cf316fde927f"}
Jul 24 12:53:39.057296 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.057264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98xw\" (UniqueName: \"kubernetes.io/projected/a6c76152-efac-4a76-bd1c-8d6afa6e105f-kube-api-access-b98xw\") pod \"payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb\" (UID: \"a6c76152-efac-4a76-bd1c-8d6afa6e105f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"
Jul 24 12:53:39.057440 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.057426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpff\" (UniqueName: \"kubernetes.io/projected/7bc085cb-4182-428a-925b-4c6306b1c2d7-kube-api-access-hwpff\") pod \"payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk\" (UID: \"7bc085cb-4182-428a-925b-4c6306b1c2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"
Jul 24 12:53:39.057519 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.057452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc085cb-4182-428a-925b-4c6306b1c2d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk\" (UID: \"7bc085cb-4182-428a-925b-4c6306b1c2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"
Jul 24 12:53:39.057519 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.057487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c76152-efac-4a76-bd1c-8d6afa6e105f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb\" (UID: \"a6c76152-efac-4a76-bd1c-8d6afa6e105f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"
Jul 24 12:53:39.058196 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.058175    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc085cb-4182-428a-925b-4c6306b1c2d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk\" (UID: \"7bc085cb-4182-428a-925b-4c6306b1c2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"
Jul 24 12:53:39.058196 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.058192    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c76152-efac-4a76-bd1c-8d6afa6e105f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb\" (UID: \"a6c76152-efac-4a76-bd1c-8d6afa6e105f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"
Jul 24 12:53:39.066283 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.066255    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpff\" (UniqueName: \"kubernetes.io/projected/7bc085cb-4182-428a-925b-4c6306b1c2d7-kube-api-access-hwpff\") pod \"payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk\" (UID: \"7bc085cb-4182-428a-925b-4c6306b1c2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"
Jul 24 12:53:39.066641 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.066624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98xw\" (UniqueName: \"kubernetes.io/projected/a6c76152-efac-4a76-bd1c-8d6afa6e105f-kube-api-access-b98xw\") pod \"payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb\" (UID: \"a6c76152-efac-4a76-bd1c-8d6afa6e105f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"
Jul 24 12:53:39.134830 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.134797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"
Jul 24 12:53:39.188385 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.188345    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"
Jul 24 12:53:39.491342 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.491307    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"]
Jul 24 12:53:39.492351 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:53:39.492327    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc085cb_4182_428a_925b_4c6306b1c2d7.slice/crio-0367af6d98045af6eafb2abd6c8561be8efbd213e178d81ba63d6acec3a292e4 WatchSource:0}: Error finding container 0367af6d98045af6eafb2abd6c8561be8efbd213e178d81ba63d6acec3a292e4: Status 404 returned error can't find the container with id 0367af6d98045af6eafb2abd6c8561be8efbd213e178d81ba63d6acec3a292e4
Jul 24 12:53:39.553482 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:39.553462    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"]
Jul 24 12:53:39.555684 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:53:39.555659    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c76152_efac_4a76_bd1c_8d6afa6e105f.slice/crio-c558958e41fd60e93f7ea5147bfec93165030bb57ad65fb04787395a6a60f6a8 WatchSource:0}: Error finding container c558958e41fd60e93f7ea5147bfec93165030bb57ad65fb04787395a6a60f6a8: Status 404 returned error can't find the container with id c558958e41fd60e93f7ea5147bfec93165030bb57ad65fb04787395a6a60f6a8
Jul 24 12:53:40.054260 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:40.054224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr" event={"ID":"e5a71b66-9be8-4010-b03c-375719b4d492","Type":"ContainerStarted","Data":"ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc"}
Jul 24 12:53:40.054701 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:40.054309    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:40.056080 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:40.056052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb" event={"ID":"a6c76152-efac-4a76-bd1c-8d6afa6e105f","Type":"ContainerStarted","Data":"e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe"}
Jul 24 12:53:40.056253 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:40.056098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb" event={"ID":"a6c76152-efac-4a76-bd1c-8d6afa6e105f","Type":"ContainerStarted","Data":"c558958e41fd60e93f7ea5147bfec93165030bb57ad65fb04787395a6a60f6a8"}
Jul 24 12:53:40.056253 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:40.056153    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"
Jul 24 12:53:40.057634 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:40.057613    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk" event={"ID":"7bc085cb-4182-428a-925b-4c6306b1c2d7","Type":"ContainerStarted","Data":"639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7"}
Jul 24 12:53:40.057748 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:40.057657    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"
Jul 24 12:53:40.057748 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:40.057679    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk" event={"ID":"7bc085cb-4182-428a-925b-4c6306b1c2d7","Type":"ContainerStarted","Data":"0367af6d98045af6eafb2abd6c8561be8efbd213e178d81ba63d6acec3a292e4"}
Jul 24 12:53:40.073843 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:40.073797    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr" podStartSLOduration=1.616007097 podStartE2EDuration="2.073781512s" totalImagesPullingTime="457.774415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:40.072135548 +0000 UTC m=+2093.599882607" watchObservedRunningTime="2026-07-24 12:53:40.073781512 +0000 UTC m=+2093.601528567"
Jul 24 12:53:40.091504 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:40.091458    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb" podStartSLOduration=2.091445944 podStartE2EDuration="2.091445944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:40.090048959 +0000 UTC m=+2093.617796014" watchObservedRunningTime="2026-07-24 12:53:40.091445944 +0000 UTC m=+2093.619192996"
Jul 24 12:53:40.108289 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:40.108248    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk" podStartSLOduration=2.108235455 podStartE2EDuration="2.108235455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:40.107288052 +0000 UTC m=+2093.635035105" watchObservedRunningTime="2026-07-24 12:53:40.108235455 +0000 UTC m=+2093.635982508"
Jul 24 12:53:42.365560 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:42.365527    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:42.724814 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:42.724725    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:43.102757 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:43.102708    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:43.345796 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:43.345761    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:45.881791 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:45.881736    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:45.960194 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:45.960149    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"]
Jul 24 12:53:45.964173 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:45.964138    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:45.965589 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:45.965572    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:45.969117 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:45.969097    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fc0bc9-serving-cert\""
Jul 24 12:53:45.974953 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:45.974597    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"]
Jul 24 12:53:46.036825 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.036789    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:46.068399 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.068363    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:53:46.120325 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.120291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr\" (UID: \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:46.120513 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.120391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr\" (UID: \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:46.120513 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.120467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nh4m\" (UniqueName: \"kubernetes.io/projected/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-kube-api-access-7nh4m\") pod \"maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr\" (UID: \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:46.221904 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.221807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr\" (UID: \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:46.222134 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.221927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nh4m\" (UniqueName: \"kubernetes.io/projected/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-kube-api-access-7nh4m\") pod \"maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr\" (UID: \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:46.222134 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.221994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr\" (UID: \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:46.222732 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.222652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr\" (UID: \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:46.224500 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.224469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr\" (UID: \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:46.231421 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.231401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nh4m\" (UniqueName: \"kubernetes.io/projected/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-kube-api-access-7nh4m\") pod \"maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr\" (UID: \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:46.246510 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.246479    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:46.278311 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.278287    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:46.428474 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:53:46.428434    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f72507_ab3d_4a89_a2ac_bf8b4f69ec5e.slice/crio-1ef4927a403fa80d1ccf694d5e9c7fcf1eb07aa96f42bb96021323b55ba71300 WatchSource:0}: Error finding container 1ef4927a403fa80d1ccf694d5e9c7fcf1eb07aa96f42bb96021323b55ba71300: Status 404 returned error can't find the container with id 1ef4927a403fa80d1ccf694d5e9c7fcf1eb07aa96f42bb96021323b55ba71300
Jul 24 12:53:46.428855 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.428827    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"]
Jul 24 12:53:46.567892 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.567857    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:46.667787 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.667748    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:46.668989 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.668967    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"]
Jul 24 12:53:46.673730 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.673716    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"
Jul 24 12:53:46.676658 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.676636    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fc0bc9\""
Jul 24 12:53:46.676751 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.676712    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fc0bc9-dockercfg-5dbj8\""
Jul 24 12:53:46.685279 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.685255    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"]
Jul 24 12:53:46.705576 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.705549    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"]
Jul 24 12:53:46.709046 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.708998    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"
Jul 24 12:53:46.721885 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.721865    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"]
Jul 24 12:53:46.828106 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.828080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786499ab-9638-4f04-b6e2-9ea995bd0490-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd\" (UID: \"786499ab-9638-4f04-b6e2-9ea995bd0490\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"
Jul 24 12:53:46.828212 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.828166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7dxn\" (UniqueName: \"kubernetes.io/projected/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936-kube-api-access-z7dxn\") pod \"payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn\" (UID: \"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"
Jul 24 12:53:46.828212 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.828188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn\" (UID: \"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"
Jul 24 12:53:46.828282 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.828218    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrdl\" (UniqueName: \"kubernetes.io/projected/786499ab-9638-4f04-b6e2-9ea995bd0490-kube-api-access-7wrdl\") pod \"payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd\" (UID: \"786499ab-9638-4f04-b6e2-9ea995bd0490\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"
Jul 24 12:53:46.879052 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.878984    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:46.928765 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.928735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786499ab-9638-4f04-b6e2-9ea995bd0490-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd\" (UID: \"786499ab-9638-4f04-b6e2-9ea995bd0490\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"
Jul 24 12:53:46.929222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.928813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7dxn\" (UniqueName: \"kubernetes.io/projected/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936-kube-api-access-z7dxn\") pod \"payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn\" (UID: \"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"
Jul 24 12:53:46.929222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.928845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn\" (UID: \"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"
Jul 24 12:53:46.929222 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.928885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrdl\" (UniqueName: \"kubernetes.io/projected/786499ab-9638-4f04-b6e2-9ea995bd0490-kube-api-access-7wrdl\") pod \"payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd\" (UID: \"786499ab-9638-4f04-b6e2-9ea995bd0490\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"
Jul 24 12:53:46.931979 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.931959    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fc0bc9\""
Jul 24 12:53:46.937651 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.937627    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrdl\" (UniqueName: \"kubernetes.io/projected/786499ab-9638-4f04-b6e2-9ea995bd0490-kube-api-access-7wrdl\") pod \"payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd\" (UID: \"786499ab-9638-4f04-b6e2-9ea995bd0490\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"
Jul 24 12:53:46.937730 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.937681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7dxn\" (UniqueName: \"kubernetes.io/projected/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936-kube-api-access-z7dxn\") pod \"payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn\" (UID: \"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"
Jul 24 12:53:46.939864 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.939849    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786499ab-9638-4f04-b6e2-9ea995bd0490-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd\" (UID: \"786499ab-9638-4f04-b6e2-9ea995bd0490\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"
Jul 24 12:53:46.939864 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.939858    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn\" (UID: \"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"
Jul 24 12:53:46.988234 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.988209    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fc0bc9-dockercfg-5dbj8\""
Jul 24 12:53:46.995783 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:46.995750    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"
Jul 24 12:53:47.019979 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:47.019954    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"
Jul 24 12:53:47.091608 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:47.091581    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:53:47.096505 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:47.096483    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:53:47.102044 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:47.101990    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr" event={"ID":"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e","Type":"ContainerStarted","Data":"6de401c86fe0a627d96a9664347d19be24ec11664697e06fa9ed9c6cf737a20a"}
Jul 24 12:53:47.102449 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:47.102332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr" event={"ID":"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e","Type":"ContainerStarted","Data":"1ef4927a403fa80d1ccf694d5e9c7fcf1eb07aa96f42bb96021323b55ba71300"}
Jul 24 12:53:47.102690 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:47.102657    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:47.132553 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:47.132500    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr" podStartSLOduration=1.8088773489999999 podStartE2EDuration="2.132477864s" totalImagesPullingTime="323.600515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:47.128899826 +0000 UTC m=+2100.656646880" watchObservedRunningTime="2026-07-24 12:53:47.132477864 +0000 UTC m=+2100.660224919"
Jul 24 12:53:47.167251 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:47.167219    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"]
Jul 24 12:53:47.167923 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:53:47.167886    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c4f61a_c9ed_4dfb_bf94_81ddbe4c0936.slice/crio-51f45eef7b60ae723d33eb68f25d8667c62a9f16ab5227a9dac9aeb505594655 WatchSource:0}: Error finding container 51f45eef7b60ae723d33eb68f25d8667c62a9f16ab5227a9dac9aeb505594655: Status 404 returned error can't find the container with id 51f45eef7b60ae723d33eb68f25d8667c62a9f16ab5227a9dac9aeb505594655
Jul 24 12:53:47.397599 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:47.397569    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"]
Jul 24 12:53:47.399459 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:53:47.399434    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786499ab_9638_4f04_b6e2_9ea995bd0490.slice/crio-091bd52d52c0aa68da156a1c148f39f1a6f6498e44c70e03b12fd3b143cdff98 WatchSource:0}: Error finding container 091bd52d52c0aa68da156a1c148f39f1a6f6498e44c70e03b12fd3b143cdff98: Status 404 returned error can't find the container with id 091bd52d52c0aa68da156a1c148f39f1a6f6498e44c70e03b12fd3b143cdff98
Jul 24 12:53:48.109263 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:48.109225    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd" event={"ID":"786499ab-9638-4f04-b6e2-9ea995bd0490","Type":"ContainerStarted","Data":"b6a0a5904c6fa593e794958a3dcc029fa161ac7dfc32931e525d980c0a8febcd"}
Jul 24 12:53:48.109748 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:48.109376    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"
Jul 24 12:53:48.109748 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:48.109416    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd" event={"ID":"786499ab-9638-4f04-b6e2-9ea995bd0490","Type":"ContainerStarted","Data":"091bd52d52c0aa68da156a1c148f39f1a6f6498e44c70e03b12fd3b143cdff98"}
Jul 24 12:53:48.110957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:48.110932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn" event={"ID":"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936","Type":"ContainerStarted","Data":"5fee6a5172e58e17c9fbcd07e0fcd56ed10073e6f808b442c937260efca83bd7"}
Jul 24 12:53:48.111106 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:48.110973    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn" event={"ID":"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936","Type":"ContainerStarted","Data":"51f45eef7b60ae723d33eb68f25d8667c62a9f16ab5227a9dac9aeb505594655"}
Jul 24 12:53:48.111106 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:48.111046    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"
Jul 24 12:53:48.126168 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:48.126124    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd" podStartSLOduration=2.126110993 podStartE2EDuration="2.126110993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:48.124621023 +0000 UTC m=+2101.652368077" watchObservedRunningTime="2026-07-24 12:53:48.126110993 +0000 UTC m=+2101.653858088"
Jul 24 12:53:48.141794 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:48.141740    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn" podStartSLOduration=2.141721548 podStartE2EDuration="2.141721548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:48.139641233 +0000 UTC m=+2101.667388287" watchObservedRunningTime="2026-07-24 12:53:48.141721548 +0000 UTC m=+2101.669468603"
Jul 24 12:53:50.369445 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:50.369411    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:53:51.065455 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:51.065425    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"
Jul 24 12:53:51.065682 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:51.065558    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"
Jul 24 12:53:53.117008 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:53.116980    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:53:59.117312 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:59.117283    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"
Jul 24 12:53:59.117734 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:53:59.117348    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"
Jul 24 12:54:03.920765 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:54:03.920725    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:54:13.186823 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:54:13.186788    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:54:13.243696 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:54:13.243655    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:54:27.252410 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:54:27.252375    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:20.397344 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:20.397312    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:21.291611 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:21.291578    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:31.961846 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:31.961271    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"]
Jul 24 12:55:31.961846 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:31.961490    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr" podUID="e5a71b66-9be8-4010-b03c-375719b4d492" containerName="maas-api" containerID="cri-o://ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc" gracePeriod=30
Jul 24 12:55:31.994272 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:31.994227    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:32.029102 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.029075    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"]
Jul 24 12:55:32.029338 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.029306    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk" podUID="7bc085cb-4182-428a-925b-4c6306b1c2d7" containerName="payload-processing" containerID="cri-o://639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7" gracePeriod=30
Jul 24 12:55:32.037836 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.037712    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"]
Jul 24 12:55:32.038292 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.037908    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb" podUID="a6c76152-efac-4a76-bd1c-8d6afa6e105f" containerName="payload-pre-processing" containerID="cri-o://e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe" gracePeriod=30
Jul 24 12:55:32.074242 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.074201    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:32.138616 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.138590    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:32.309691 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.309594    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"
Jul 24 12:55:32.352092 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.352067    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"
Jul 24 12:55:32.414543 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.414522    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:55:32.424797 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.424776    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc085cb-4182-428a-925b-4c6306b1c2d7-plugins-config-volume\") pod \"7bc085cb-4182-428a-925b-4c6306b1c2d7\" (UID: \"7bc085cb-4182-428a-925b-4c6306b1c2d7\") "
Jul 24 12:55:32.424911 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.424899    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwpff\" (UniqueName: \"kubernetes.io/projected/7bc085cb-4182-428a-925b-4c6306b1c2d7-kube-api-access-hwpff\") pod \"7bc085cb-4182-428a-925b-4c6306b1c2d7\" (UID: \"7bc085cb-4182-428a-925b-4c6306b1c2d7\") "
Jul 24 12:55:32.425197 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.425172    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc085cb-4182-428a-925b-4c6306b1c2d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7bc085cb-4182-428a-925b-4c6306b1c2d7" (UID: "7bc085cb-4182-428a-925b-4c6306b1c2d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:32.426908 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.426892    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc085cb-4182-428a-925b-4c6306b1c2d7-kube-api-access-hwpff" (OuterVolumeSpecName: "kube-api-access-hwpff") pod "7bc085cb-4182-428a-925b-4c6306b1c2d7" (UID: "7bc085cb-4182-428a-925b-4c6306b1c2d7"). InnerVolumeSpecName "kube-api-access-hwpff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:55:32.525717 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.525694    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a71b66-9be8-4010-b03c-375719b4d492-openshift-service-ca\") pod \"e5a71b66-9be8-4010-b03c-375719b4d492\" (UID: \"e5a71b66-9be8-4010-b03c-375719b4d492\") "
Jul 24 12:55:32.525876 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.525772    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c76152-efac-4a76-bd1c-8d6afa6e105f-plugins-config-volume\") pod \"a6c76152-efac-4a76-bd1c-8d6afa6e105f\" (UID: \"a6c76152-efac-4a76-bd1c-8d6afa6e105f\") "
Jul 24 12:55:32.525876 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.525800    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5a71b66-9be8-4010-b03c-375719b4d492-maas-api-tls\") pod \"e5a71b66-9be8-4010-b03c-375719b4d492\" (UID: \"e5a71b66-9be8-4010-b03c-375719b4d492\") "
Jul 24 12:55:32.525876 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.525845    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pcz8\" (UniqueName: \"kubernetes.io/projected/e5a71b66-9be8-4010-b03c-375719b4d492-kube-api-access-2pcz8\") pod \"e5a71b66-9be8-4010-b03c-375719b4d492\" (UID: \"e5a71b66-9be8-4010-b03c-375719b4d492\") "
Jul 24 12:55:32.526151 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.525895    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b98xw\" (UniqueName: \"kubernetes.io/projected/a6c76152-efac-4a76-bd1c-8d6afa6e105f-kube-api-access-b98xw\") pod \"a6c76152-efac-4a76-bd1c-8d6afa6e105f\" (UID: \"a6c76152-efac-4a76-bd1c-8d6afa6e105f\") "
Jul 24 12:55:32.526151 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.526101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a71b66-9be8-4010-b03c-375719b4d492-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5a71b66-9be8-4010-b03c-375719b4d492" (UID: "e5a71b66-9be8-4010-b03c-375719b4d492"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:32.526258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.526215    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwpff\" (UniqueName: \"kubernetes.io/projected/7bc085cb-4182-428a-925b-4c6306b1c2d7-kube-api-access-hwpff\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:32.526258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.526211    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c76152-efac-4a76-bd1c-8d6afa6e105f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a6c76152-efac-4a76-bd1c-8d6afa6e105f" (UID: "a6c76152-efac-4a76-bd1c-8d6afa6e105f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:32.526258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.526231    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a71b66-9be8-4010-b03c-375719b4d492-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:32.526258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.526253    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc085cb-4182-428a-925b-4c6306b1c2d7-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:32.528042 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.528004    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5a71b66-9be8-4010-b03c-375719b4d492-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5a71b66-9be8-4010-b03c-375719b4d492" (UID: "e5a71b66-9be8-4010-b03c-375719b4d492"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:32.528114 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.528098    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a71b66-9be8-4010-b03c-375719b4d492-kube-api-access-2pcz8" (OuterVolumeSpecName: "kube-api-access-2pcz8") pod "e5a71b66-9be8-4010-b03c-375719b4d492" (UID: "e5a71b66-9be8-4010-b03c-375719b4d492"). InnerVolumeSpecName "kube-api-access-2pcz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:55:32.528152 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.528125    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c76152-efac-4a76-bd1c-8d6afa6e105f-kube-api-access-b98xw" (OuterVolumeSpecName: "kube-api-access-b98xw") pod "a6c76152-efac-4a76-bd1c-8d6afa6e105f" (UID: "a6c76152-efac-4a76-bd1c-8d6afa6e105f"). InnerVolumeSpecName "kube-api-access-b98xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:55:32.624769 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.624737    2566 generic.go:356] "Generic (PLEG): container finished" podID="e5a71b66-9be8-4010-b03c-375719b4d492" containerID="ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc" exitCode=0
Jul 24 12:55:32.624868 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.624786    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"
Jul 24 12:55:32.624868 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.624826    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr" event={"ID":"e5a71b66-9be8-4010-b03c-375719b4d492","Type":"ContainerDied","Data":"ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc"}
Jul 24 12:55:32.624945 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.624902    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr" event={"ID":"e5a71b66-9be8-4010-b03c-375719b4d492","Type":"ContainerDied","Data":"2dc8e8c3e612c4d02a62e2e14a56e4629c028057068ab5fd3113cf316fde927f"}
Jul 24 12:55:32.624945 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.624929    2566 scope.go:117] "RemoveContainer" containerID="ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc"
Jul 24 12:55:32.626106 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.626085    2566 generic.go:356] "Generic (PLEG): container finished" podID="a6c76152-efac-4a76-bd1c-8d6afa6e105f" containerID="e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe" exitCode=0
Jul 24 12:55:32.626211 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.626133    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"
Jul 24 12:55:32.626211 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.626169    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb" event={"ID":"a6c76152-efac-4a76-bd1c-8d6afa6e105f","Type":"ContainerDied","Data":"e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe"}
Jul 24 12:55:32.626302 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.626218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb" event={"ID":"a6c76152-efac-4a76-bd1c-8d6afa6e105f","Type":"ContainerDied","Data":"c558958e41fd60e93f7ea5147bfec93165030bb57ad65fb04787395a6a60f6a8"}
Jul 24 12:55:32.626680 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.626659    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pcz8\" (UniqueName: \"kubernetes.io/projected/e5a71b66-9be8-4010-b03c-375719b4d492-kube-api-access-2pcz8\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:32.626773 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.626684    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b98xw\" (UniqueName: \"kubernetes.io/projected/a6c76152-efac-4a76-bd1c-8d6afa6e105f-kube-api-access-b98xw\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:32.626773 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.626699    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c76152-efac-4a76-bd1c-8d6afa6e105f-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:32.626773 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.626711    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5a71b66-9be8-4010-b03c-375719b4d492-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:32.627570 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.627544    2566 generic.go:356] "Generic (PLEG): container finished" podID="7bc085cb-4182-428a-925b-4c6306b1c2d7" containerID="639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7" exitCode=0
Jul 24 12:55:32.627673 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.627574    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"
Jul 24 12:55:32.627673 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.627621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk" event={"ID":"7bc085cb-4182-428a-925b-4c6306b1c2d7","Type":"ContainerDied","Data":"639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7"}
Jul 24 12:55:32.627772 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.627681    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk" event={"ID":"7bc085cb-4182-428a-925b-4c6306b1c2d7","Type":"ContainerDied","Data":"0367af6d98045af6eafb2abd6c8561be8efbd213e178d81ba63d6acec3a292e4"}
Jul 24 12:55:32.634888 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.634871    2566 scope.go:117] "RemoveContainer" containerID="ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc"
Jul 24 12:55:32.635248 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:55:32.635187    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc\": container with ID starting with ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc not found: ID does not exist" containerID="ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc"
Jul 24 12:55:32.635248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.635224    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc"} err="failed to get container status \"ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc\": rpc error: code = NotFound desc = could not find container \"ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc\": container with ID starting with ed19dc9d1fc6bdaaa930c0ad8187e1c457bb9ccb4e522a4e0069ede625da01cc not found: ID does not exist"
Jul 24 12:55:32.635248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.635246    2566 scope.go:117] "RemoveContainer" containerID="e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe"
Jul 24 12:55:32.643953 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.643938    2566 scope.go:117] "RemoveContainer" containerID="e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe"
Jul 24 12:55:32.644206 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:55:32.644191    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe\": container with ID starting with e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe not found: ID does not exist" containerID="e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe"
Jul 24 12:55:32.644267 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.644211    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe"} err="failed to get container status \"e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe\": rpc error: code = NotFound desc = could not find container \"e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe\": container with ID starting with e4d9a7422015a34f6cc9d8eac50b125b7055ceb02b7f059aa26d253a445a2afe not found: ID does not exist"
Jul 24 12:55:32.644267 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.644226    2566 scope.go:117] "RemoveContainer" containerID="639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7"
Jul 24 12:55:32.651638 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.651606    2566 scope.go:117] "RemoveContainer" containerID="639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7"
Jul 24 12:55:32.651859 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:55:32.651842    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7\": container with ID starting with 639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7 not found: ID does not exist" containerID="639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7"
Jul 24 12:55:32.651926 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.651869    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7"} err="failed to get container status \"639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7\": rpc error: code = NotFound desc = could not find container \"639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7\": container with ID starting with 639d9c2bd58b4a6c71cef3c22647c9fce6ce8694d5ed2988604a5a6075b4e6c7 not found: ID does not exist"
Jul 24 12:55:32.665211 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.665183    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"]
Jul 24 12:55:32.668555 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.668534    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd4baf-7fc4bb7ddb-kmzvr"]
Jul 24 12:55:32.687219 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.687198    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"]
Jul 24 12:55:32.690412 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.690394    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dd4baf-976f7978d-gqvdb"]
Jul 24 12:55:32.710399 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.710378    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"]
Jul 24 12:55:32.713461 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.713442    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dd4baf-5fb856d6d-vlqqk"]
Jul 24 12:55:32.976379 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.976314    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc085cb-4182-428a-925b-4c6306b1c2d7" path="/var/lib/kubelet/pods/7bc085cb-4182-428a-925b-4c6306b1c2d7/volumes"
Jul 24 12:55:32.976688 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.976650    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6c76152-efac-4a76-bd1c-8d6afa6e105f" path="/var/lib/kubelet/pods/a6c76152-efac-4a76-bd1c-8d6afa6e105f/volumes"
Jul 24 12:55:32.976973 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:32.976960    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5a71b66-9be8-4010-b03c-375719b4d492" path="/var/lib/kubelet/pods/e5a71b66-9be8-4010-b03c-375719b4d492/volumes"
Jul 24 12:55:36.541256 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:36.541218    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:37.015315 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:37.015276    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:43.152670 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:43.152637    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:43.257707 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:43.257678    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:43.476771 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:43.476685    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:43.733586 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:43.733505    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:54.420042 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.419979    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"]
Jul 24 12:55:54.420430 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.420257    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr" podUID="c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e" containerName="maas-api" containerID="cri-o://6de401c86fe0a627d96a9664347d19be24ec11664697e06fa9ed9c6cf737a20a" gracePeriod=30
Jul 24 12:55:54.461050 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.460997    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:54.490596 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.490568    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"]
Jul 24 12:55:54.490885 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.490835    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd" podUID="786499ab-9638-4f04-b6e2-9ea995bd0490" containerName="payload-processing" containerID="cri-o://b6a0a5904c6fa593e794958a3dcc029fa161ac7dfc32931e525d980c0a8febcd" gracePeriod=30
Jul 24 12:55:54.493424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.493391    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"]
Jul 24 12:55:54.493604 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.493584    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn" podUID="e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936" containerName="payload-pre-processing" containerID="cri-o://5fee6a5172e58e17c9fbcd07e0fcd56ed10073e6f808b442c937260efca83bd7" gracePeriod=30
Jul 24 12:55:54.574564 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.574535    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:54.670814 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.670697    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:55:54.741281 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.741241    2566 generic.go:356] "Generic (PLEG): container finished" podID="786499ab-9638-4f04-b6e2-9ea995bd0490" containerID="b6a0a5904c6fa593e794958a3dcc029fa161ac7dfc32931e525d980c0a8febcd" exitCode=0
Jul 24 12:55:54.741428 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.741409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd" event={"ID":"786499ab-9638-4f04-b6e2-9ea995bd0490","Type":"ContainerDied","Data":"b6a0a5904c6fa593e794958a3dcc029fa161ac7dfc32931e525d980c0a8febcd"}
Jul 24 12:55:54.744285 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.744255    2566 generic.go:356] "Generic (PLEG): container finished" podID="c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e" containerID="6de401c86fe0a627d96a9664347d19be24ec11664697e06fa9ed9c6cf737a20a" exitCode=0
Jul 24 12:55:54.744471 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.744376    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr" event={"ID":"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e","Type":"ContainerDied","Data":"6de401c86fe0a627d96a9664347d19be24ec11664697e06fa9ed9c6cf737a20a"}
Jul 24 12:55:54.746208 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.746183    2566 generic.go:356] "Generic (PLEG): container finished" podID="e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936" containerID="5fee6a5172e58e17c9fbcd07e0fcd56ed10073e6f808b442c937260efca83bd7" exitCode=0
Jul 24 12:55:54.746294 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.746218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn" event={"ID":"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936","Type":"ContainerDied","Data":"5fee6a5172e58e17c9fbcd07e0fcd56ed10073e6f808b442c937260efca83bd7"}
Jul 24 12:55:54.776624 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.776595    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"
Jul 24 12:55:54.798839 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.798816    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:55:54.800862 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.800322    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786499ab-9638-4f04-b6e2-9ea995bd0490-plugins-config-volume\") pod \"786499ab-9638-4f04-b6e2-9ea995bd0490\" (UID: \"786499ab-9638-4f04-b6e2-9ea995bd0490\") "
Jul 24 12:55:54.800862 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.800433    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wrdl\" (UniqueName: \"kubernetes.io/projected/786499ab-9638-4f04-b6e2-9ea995bd0490-kube-api-access-7wrdl\") pod \"786499ab-9638-4f04-b6e2-9ea995bd0490\" (UID: \"786499ab-9638-4f04-b6e2-9ea995bd0490\") "
Jul 24 12:55:54.800862 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.800693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786499ab-9638-4f04-b6e2-9ea995bd0490-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "786499ab-9638-4f04-b6e2-9ea995bd0490" (UID: "786499ab-9638-4f04-b6e2-9ea995bd0490"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:54.806289 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.806265    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786499ab-9638-4f04-b6e2-9ea995bd0490-kube-api-access-7wrdl" (OuterVolumeSpecName: "kube-api-access-7wrdl") pod "786499ab-9638-4f04-b6e2-9ea995bd0490" (UID: "786499ab-9638-4f04-b6e2-9ea995bd0490"). InnerVolumeSpecName "kube-api-access-7wrdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:55:54.810220 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.810195    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"
Jul 24 12:55:54.901235 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.901208    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-maas-api-tls\") pod \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\" (UID: \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\") "
Jul 24 12:55:54.901389 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.901246    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nh4m\" (UniqueName: \"kubernetes.io/projected/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-kube-api-access-7nh4m\") pod \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\" (UID: \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\") "
Jul 24 12:55:54.901389 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.901311    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-openshift-service-ca\") pod \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\" (UID: \"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e\") "
Jul 24 12:55:54.901389 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.901355    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7dxn\" (UniqueName: \"kubernetes.io/projected/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936-kube-api-access-z7dxn\") pod \"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936\" (UID: \"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936\") "
Jul 24 12:55:54.901389 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.901377    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936-plugins-config-volume\") pod \"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936\" (UID: \"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936\") "
Jul 24 12:55:54.901697 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.901653    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e" (UID: "c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:54.901832 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.901734    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936" (UID: "e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:54.901832 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.901814    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:54.902139 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.901834    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wrdl\" (UniqueName: \"kubernetes.io/projected/786499ab-9638-4f04-b6e2-9ea995bd0490-kube-api-access-7wrdl\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:54.902139 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.901849    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786499ab-9638-4f04-b6e2-9ea995bd0490-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:54.903485 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.903464    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e" (UID: "c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:54.903485 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.903468    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-kube-api-access-7nh4m" (OuterVolumeSpecName: "kube-api-access-7nh4m") pod "c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e" (UID: "c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e"). InnerVolumeSpecName "kube-api-access-7nh4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:55:54.903603 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:54.903506    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936-kube-api-access-z7dxn" (OuterVolumeSpecName: "kube-api-access-z7dxn") pod "e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936" (UID: "e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936"). InnerVolumeSpecName "kube-api-access-z7dxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:55:55.003085 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.003065    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:55.003085 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.003083    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nh4m\" (UniqueName: \"kubernetes.io/projected/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e-kube-api-access-7nh4m\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:55.003206 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.003092    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7dxn\" (UniqueName: \"kubernetes.io/projected/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936-kube-api-access-z7dxn\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:55.003206 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.003102    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:55:55.752897 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.752871    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"
Jul 24 12:55:55.752897 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.752885    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr" event={"ID":"c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e","Type":"ContainerDied","Data":"1ef4927a403fa80d1ccf694d5e9c7fcf1eb07aa96f42bb96021323b55ba71300"}
Jul 24 12:55:55.753435 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.752949    2566 scope.go:117] "RemoveContainer" containerID="6de401c86fe0a627d96a9664347d19be24ec11664697e06fa9ed9c6cf737a20a"
Jul 24 12:55:55.754340 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.754163    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"
Jul 24 12:55:55.754340 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.754211    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn" event={"ID":"e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936","Type":"ContainerDied","Data":"51f45eef7b60ae723d33eb68f25d8667c62a9f16ab5227a9dac9aeb505594655"}
Jul 24 12:55:55.755789 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.755769    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"
Jul 24 12:55:55.755898 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.755794    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd" event={"ID":"786499ab-9638-4f04-b6e2-9ea995bd0490","Type":"ContainerDied","Data":"091bd52d52c0aa68da156a1c148f39f1a6f6498e44c70e03b12fd3b143cdff98"}
Jul 24 12:55:55.762442 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.762424    2566 scope.go:117] "RemoveContainer" containerID="5fee6a5172e58e17c9fbcd07e0fcd56ed10073e6f808b442c937260efca83bd7"
Jul 24 12:55:55.769711 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.769695    2566 scope.go:117] "RemoveContainer" containerID="b6a0a5904c6fa593e794958a3dcc029fa161ac7dfc32931e525d980c0a8febcd"
Jul 24 12:55:55.773316 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.773297    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"]
Jul 24 12:55:55.777267 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.777247    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc0bc9-5dbc9dd7b9-gmpdr"]
Jul 24 12:55:55.797141 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.797116    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"]
Jul 24 12:55:55.800396 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.800377    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fc0bc9-f547b9b84-zgdjn"]
Jul 24 12:55:55.814952 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.814933    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"]
Jul 24 12:55:55.818599 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:55.818581    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fc0bc9-57d5686b-x6hgd"]
Jul 24 12:55:56.974569 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:56.974530    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="786499ab-9638-4f04-b6e2-9ea995bd0490" path="/var/lib/kubelet/pods/786499ab-9638-4f04-b6e2-9ea995bd0490/volumes"
Jul 24 12:55:56.975065 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:56.975052    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e" path="/var/lib/kubelet/pods/c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e/volumes"
Jul 24 12:55:56.975424 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:56.975404    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936" path="/var/lib/kubelet/pods/e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936/volumes"
Jul 24 12:55:58.917521 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:55:58.917480    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:00.276075 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:00.276040    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:05.219571 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:05.219539    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:05.316372 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:05.316337    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:05.530875 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:05.530836    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:05.761938 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:05.761894    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:06.125870 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:06.125832    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:06.481069 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:06.480960    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:06.600970 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:06.600940    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:09.783677 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.783594    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:09.911564 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.911492    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:09.947862 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.947829    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"]
Jul 24 12:56:09.948215 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948203    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a71b66-9be8-4010-b03c-375719b4d492" containerName="maas-api"
Jul 24 12:56:09.948270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948217    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a71b66-9be8-4010-b03c-375719b4d492" containerName="maas-api"
Jul 24 12:56:09.948270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948230    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936" containerName="payload-pre-processing"
Jul 24 12:56:09.948270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948236    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936" containerName="payload-pre-processing"
Jul 24 12:56:09.948270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948254    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e" containerName="maas-api"
Jul 24 12:56:09.948270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948259    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e" containerName="maas-api"
Jul 24 12:56:09.948270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948266    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="786499ab-9638-4f04-b6e2-9ea995bd0490" containerName="payload-processing"
Jul 24 12:56:09.948270 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948272    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="786499ab-9638-4f04-b6e2-9ea995bd0490" containerName="payload-processing"
Jul 24 12:56:09.948507 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948279    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc085cb-4182-428a-925b-4c6306b1c2d7" containerName="payload-processing"
Jul 24 12:56:09.948507 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948285    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc085cb-4182-428a-925b-4c6306b1c2d7" containerName="payload-processing"
Jul 24 12:56:09.948507 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948290    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c76152-efac-4a76-bd1c-8d6afa6e105f" containerName="payload-pre-processing"
Jul 24 12:56:09.948507 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948295    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c76152-efac-4a76-bd1c-8d6afa6e105f" containerName="payload-pre-processing"
Jul 24 12:56:09.948507 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948352    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9f72507-ab3d-4a89-a2ac-bf8b4f69ec5e" containerName="maas-api"
Jul 24 12:56:09.948507 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948360    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc085cb-4182-428a-925b-4c6306b1c2d7" containerName="payload-processing"
Jul 24 12:56:09.948507 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948368    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="786499ab-9638-4f04-b6e2-9ea995bd0490" containerName="payload-processing"
Jul 24 12:56:09.948507 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948375    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6c76152-efac-4a76-bd1c-8d6afa6e105f" containerName="payload-pre-processing"
Jul 24 12:56:09.948507 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948379    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c4f61a-c9ed-4dfb-bf94-81ddbe4c0936" containerName="payload-pre-processing"
Jul 24 12:56:09.948507 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.948385    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5a71b66-9be8-4010-b03c-375719b4d492" containerName="maas-api"
Jul 24 12:56:09.953081 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.953063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:09.957191 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.957172    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f394b5-serving-cert\""
Jul 24 12:56:09.969373 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:09.969351    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"]
Jul 24 12:56:10.016443 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.016421    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:10.026530 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.026507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-openshift-service-ca\") pod \"maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q\" (UID: \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:10.026665 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.026648    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-maas-api-tls\") pod \"maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q\" (UID: \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:10.026726 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.026710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lzg7\" (UniqueName: \"kubernetes.io/projected/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-kube-api-access-8lzg7\") pod \"maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q\" (UID: \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:10.127441 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.127376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-maas-api-tls\") pod \"maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q\" (UID: \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:10.127441 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.127415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lzg7\" (UniqueName: \"kubernetes.io/projected/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-kube-api-access-8lzg7\") pod \"maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q\" (UID: \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:10.127632 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.127451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-openshift-service-ca\") pod \"maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q\" (UID: \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:10.128142 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.128123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-openshift-service-ca\") pod \"maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q\" (UID: \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:10.129922 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.129900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-maas-api-tls\") pod \"maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q\" (UID: \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:10.140699 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.140676    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:10.142356 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.142336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lzg7\" (UniqueName: \"kubernetes.io/projected/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-kube-api-access-8lzg7\") pod \"maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q\" (UID: \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:10.262863 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.262841    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:10.419824 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.419801    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"]
Jul 24 12:56:10.421784 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:56:10.421753    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a6894c_1b1b_4c44_a0b9_58cd7b946545.slice/crio-f111eef4b4ff63a098d9f31dcc1d749df2b63d94d3fffdf8a70d204f4dc77732 WatchSource:0}: Error finding container f111eef4b4ff63a098d9f31dcc1d749df2b63d94d3fffdf8a70d204f4dc77732: Status 404 returned error can't find the container with id f111eef4b4ff63a098d9f31dcc1d749df2b63d94d3fffdf8a70d204f4dc77732
Jul 24 12:56:10.423432 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.423416    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 12:56:10.665583 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.665549    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:10.794334 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.794304    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:10.833179 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.833150    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q" event={"ID":"c1a6894c-1b1b-4c44-a0b9-58cd7b946545","Type":"ContainerStarted","Data":"f111eef4b4ff63a098d9f31dcc1d749df2b63d94d3fffdf8a70d204f4dc77732"}
Jul 24 12:56:10.913861 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.913834    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"]
Jul 24 12:56:10.917304 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.917289    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"
Jul 24 12:56:10.920776 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.920755    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f394b5-dockercfg-qcpsv\""
Jul 24 12:56:10.920935 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.920918    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f394b5\""
Jul 24 12:56:10.935787 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.935747    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"]
Jul 24 12:56:10.988707 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.984244    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"]
Jul 24 12:56:10.995040 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:10.994774    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"
Jul 24 12:56:11.007043 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.003693    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"]
Jul 24 12:56:11.036379 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.036355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdp8d\" (UniqueName: \"kubernetes.io/projected/a5317ef6-e393-4d0f-ba58-d753d09421cd-kube-api-access-sdp8d\") pod \"payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2\" (UID: \"a5317ef6-e393-4d0f-ba58-d753d09421cd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"
Jul 24 12:56:11.036508 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.036474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzv4s\" (UniqueName: \"kubernetes.io/projected/bb6b407c-5f56-4c24-b070-7e0a121067a6-kube-api-access-fzv4s\") pod \"payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c\" (UID: \"bb6b407c-5f56-4c24-b070-7e0a121067a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"
Jul 24 12:56:11.036563 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.036509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5317ef6-e393-4d0f-ba58-d753d09421cd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2\" (UID: \"a5317ef6-e393-4d0f-ba58-d753d09421cd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"
Jul 24 12:56:11.036750 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.036733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6b407c-5f56-4c24-b070-7e0a121067a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c\" (UID: \"bb6b407c-5f56-4c24-b070-7e0a121067a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"
Jul 24 12:56:11.138054 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.137946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6b407c-5f56-4c24-b070-7e0a121067a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c\" (UID: \"bb6b407c-5f56-4c24-b070-7e0a121067a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"
Jul 24 12:56:11.138054 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.137995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdp8d\" (UniqueName: \"kubernetes.io/projected/a5317ef6-e393-4d0f-ba58-d753d09421cd-kube-api-access-sdp8d\") pod \"payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2\" (UID: \"a5317ef6-e393-4d0f-ba58-d753d09421cd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"
Jul 24 12:56:11.138278 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.138102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzv4s\" (UniqueName: \"kubernetes.io/projected/bb6b407c-5f56-4c24-b070-7e0a121067a6-kube-api-access-fzv4s\") pod \"payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c\" (UID: \"bb6b407c-5f56-4c24-b070-7e0a121067a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"
Jul 24 12:56:11.138278 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.138135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5317ef6-e393-4d0f-ba58-d753d09421cd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2\" (UID: \"a5317ef6-e393-4d0f-ba58-d753d09421cd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"
Jul 24 12:56:11.138598 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.138575    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6b407c-5f56-4c24-b070-7e0a121067a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c\" (UID: \"bb6b407c-5f56-4c24-b070-7e0a121067a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"
Jul 24 12:56:11.138835 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.138813    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5317ef6-e393-4d0f-ba58-d753d09421cd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2\" (UID: \"a5317ef6-e393-4d0f-ba58-d753d09421cd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"
Jul 24 12:56:11.152255 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.152234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzv4s\" (UniqueName: \"kubernetes.io/projected/bb6b407c-5f56-4c24-b070-7e0a121067a6-kube-api-access-fzv4s\") pod \"payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c\" (UID: \"bb6b407c-5f56-4c24-b070-7e0a121067a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"
Jul 24 12:56:11.152355 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.152338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdp8d\" (UniqueName: \"kubernetes.io/projected/a5317ef6-e393-4d0f-ba58-d753d09421cd-kube-api-access-sdp8d\") pod \"payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2\" (UID: \"a5317ef6-e393-4d0f-ba58-d753d09421cd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"
Jul 24 12:56:11.228400 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.228376    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"
Jul 24 12:56:11.309647 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.309615    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"
Jul 24 12:56:11.379329 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.379250    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"]
Jul 24 12:56:11.383353 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:56:11.383321    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6b407c_5f56_4c24_b070_7e0a121067a6.slice/crio-67a2656a469f303c5884a80b3ef6eb241dbc2ef7c76a66c3bfb8f5a7b64c829c WatchSource:0}: Error finding container 67a2656a469f303c5884a80b3ef6eb241dbc2ef7c76a66c3bfb8f5a7b64c829c: Status 404 returned error can't find the container with id 67a2656a469f303c5884a80b3ef6eb241dbc2ef7c76a66c3bfb8f5a7b64c829c
Jul 24 12:56:11.470005 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:56:11.469982    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5317ef6_e393_4d0f_ba58_d753d09421cd.slice/crio-32dfda05016a9bc94f92278993fc32bd9aaafad6a31d48044182456dd747bd1b WatchSource:0}: Error finding container 32dfda05016a9bc94f92278993fc32bd9aaafad6a31d48044182456dd747bd1b: Status 404 returned error can't find the container with id 32dfda05016a9bc94f92278993fc32bd9aaafad6a31d48044182456dd747bd1b
Jul 24 12:56:11.471938 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.471913    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"]
Jul 24 12:56:11.839609 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.839572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c" event={"ID":"bb6b407c-5f56-4c24-b070-7e0a121067a6","Type":"ContainerStarted","Data":"be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c"}
Jul 24 12:56:11.840069 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.839629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c" event={"ID":"bb6b407c-5f56-4c24-b070-7e0a121067a6","Type":"ContainerStarted","Data":"67a2656a469f303c5884a80b3ef6eb241dbc2ef7c76a66c3bfb8f5a7b64c829c"}
Jul 24 12:56:11.840069 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.839669    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"
Jul 24 12:56:11.841195 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.841174    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2" event={"ID":"a5317ef6-e393-4d0f-ba58-d753d09421cd","Type":"ContainerStarted","Data":"7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9"}
Jul 24 12:56:11.841318 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.841218    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"
Jul 24 12:56:11.841318 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.841239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2" event={"ID":"a5317ef6-e393-4d0f-ba58-d753d09421cd","Type":"ContainerStarted","Data":"32dfda05016a9bc94f92278993fc32bd9aaafad6a31d48044182456dd747bd1b"}
Jul 24 12:56:11.842571 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.842553    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q" event={"ID":"c1a6894c-1b1b-4c44-a0b9-58cd7b946545","Type":"ContainerStarted","Data":"5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2"}
Jul 24 12:56:11.842693 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.842677    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:11.873676 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.873638    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c" podStartSLOduration=1.873627135 podStartE2EDuration="1.873627135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:56:11.872244057 +0000 UTC m=+2245.399991101" watchObservedRunningTime="2026-07-24 12:56:11.873627135 +0000 UTC m=+2245.401374187"
Jul 24 12:56:11.893408 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.893365    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2" podStartSLOduration=1.893353067 podStartE2EDuration="1.893353067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:56:11.892639248 +0000 UTC m=+2245.420386300" watchObservedRunningTime="2026-07-24 12:56:11.893353067 +0000 UTC m=+2245.421100118"
Jul 24 12:56:11.921219 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:11.921184    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q" podStartSLOduration=2.410710748 podStartE2EDuration="2.921172284s" totalImagesPullingTime="510.461536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:56:11.919636652 +0000 UTC m=+2245.447383706" watchObservedRunningTime="2026-07-24 12:56:11.921172284 +0000 UTC m=+2245.448919336"
Jul 24 12:56:14.267717 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:14.267685    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:14.917745 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:14.917714    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:15.417288 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:15.417249    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:15.620818 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:15.620783    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:17.853624 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:17.853597    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:19.223900 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.223863    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:19.319283 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.319253    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"]
Jul 24 12:56:19.324067 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.324045    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:19.327114 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.327094    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-aa0820-serving-cert\""
Jul 24 12:56:19.358745 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.358714    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"]
Jul 24 12:56:19.407425 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.407400    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:19.409687 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.409661    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-maas-api-tls\") pod \"maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz\" (UID: \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:19.409796 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.409749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-openshift-service-ca\") pod \"maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz\" (UID: \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:19.409796 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.409777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgrtf\" (UniqueName: \"kubernetes.io/projected/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-kube-api-access-qgrtf\") pod \"maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz\" (UID: \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:19.510844 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.510819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-maas-api-tls\") pod \"maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz\" (UID: \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:19.510956 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.510892    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-openshift-service-ca\") pod \"maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz\" (UID: \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:19.510956 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.510921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgrtf\" (UniqueName: \"kubernetes.io/projected/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-kube-api-access-qgrtf\") pod \"maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz\" (UID: \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:19.511550 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.511534    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-openshift-service-ca\") pod \"maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz\" (UID: \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:19.513432 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.513413    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-maas-api-tls\") pod \"maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz\" (UID: \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:19.524231 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.524213    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgrtf\" (UniqueName: \"kubernetes.io/projected/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-kube-api-access-qgrtf\") pod \"maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz\" (UID: \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:19.574823 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.574794    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:19.637482 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.637461    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:19.801673 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.801605    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"]
Jul 24 12:56:19.804880 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:56:19.804855    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaeeee5f_55ae_4903_8b4a_8e49a1a2eb11.slice/crio-57c5f262dc5a39bcebc1aec0c57be6a550d85f1634ec81e89f965a79a82c5ae3 WatchSource:0}: Error finding container 57c5f262dc5a39bcebc1aec0c57be6a550d85f1634ec81e89f965a79a82c5ae3: Status 404 returned error can't find the container with id 57c5f262dc5a39bcebc1aec0c57be6a550d85f1634ec81e89f965a79a82c5ae3
Jul 24 12:56:19.890279 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:19.890255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz" event={"ID":"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11","Type":"ContainerStarted","Data":"57c5f262dc5a39bcebc1aec0c57be6a550d85f1634ec81e89f965a79a82c5ae3"}
Jul 24 12:56:20.435002 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.434968    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:20.637838 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.637808    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:20.694051 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.694009    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"]
Jul 24 12:56:20.697525 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.697510    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"
Jul 24 12:56:20.707343 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.707321    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-aa0820\""
Jul 24 12:56:20.731075 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.731049    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"]
Jul 24 12:56:20.822541 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.822512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b669493-012b-4988-8f1f-3a88a4cf90ce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz\" (UID: \"3b669493-012b-4988-8f1f-3a88a4cf90ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"
Jul 24 12:56:20.822677 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.822567    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wf5x\" (UniqueName: \"kubernetes.io/projected/3b669493-012b-4988-8f1f-3a88a4cf90ce-kube-api-access-9wf5x\") pod \"payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz\" (UID: \"3b669493-012b-4988-8f1f-3a88a4cf90ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"
Jul 24 12:56:20.897823 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.897744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz" event={"ID":"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11","Type":"ContainerStarted","Data":"1d2e12be94ba5b1e64e2aecb1b5991cf105b6785d0e508c5293ef680626c0e75"}
Jul 24 12:56:20.897965 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.897895    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:20.923314 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.923290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b669493-012b-4988-8f1f-3a88a4cf90ce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz\" (UID: \"3b669493-012b-4988-8f1f-3a88a4cf90ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"
Jul 24 12:56:20.923415 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.923348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wf5x\" (UniqueName: \"kubernetes.io/projected/3b669493-012b-4988-8f1f-3a88a4cf90ce-kube-api-access-9wf5x\") pod \"payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz\" (UID: \"3b669493-012b-4988-8f1f-3a88a4cf90ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"
Jul 24 12:56:20.924072 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.924055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b669493-012b-4988-8f1f-3a88a4cf90ce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz\" (UID: \"3b669493-012b-4988-8f1f-3a88a4cf90ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"
Jul 24 12:56:20.934214 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.934171    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz" podStartSLOduration=1.5043765850000002 podStartE2EDuration="1.934158771s" totalImagesPullingTime="429.782186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:56:20.933804404 +0000 UTC m=+2254.461551458" watchObservedRunningTime="2026-07-24 12:56:20.934158771 +0000 UTC m=+2254.461905823"
Jul 24 12:56:20.943725 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:20.943702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wf5x\" (UniqueName: \"kubernetes.io/projected/3b669493-012b-4988-8f1f-3a88a4cf90ce-kube-api-access-9wf5x\") pod \"payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz\" (UID: \"3b669493-012b-4988-8f1f-3a88a4cf90ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"
Jul 24 12:56:21.007953 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:21.007929    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"
Jul 24 12:56:21.169389 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:21.169365    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"]
Jul 24 12:56:21.171336 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:56:21.171307    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b669493_012b_4988_8f1f_3a88a4cf90ce.slice/crio-f2732a77c1e28bb4f50f550493925338839c9e8e51763b90e182f45d8537b420 WatchSource:0}: Error finding container f2732a77c1e28bb4f50f550493925338839c9e8e51763b90e182f45d8537b420: Status 404 returned error can't find the container with id f2732a77c1e28bb4f50f550493925338839c9e8e51763b90e182f45d8537b420
Jul 24 12:56:21.401709 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:21.401672    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:21.904186 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:21.904150    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz" event={"ID":"3b669493-012b-4988-8f1f-3a88a4cf90ce","Type":"ContainerStarted","Data":"6e3cee67cb964cb5a871b197a32ad7881fd0a4e0bd88e1574cc1102eb343af87"}
Jul 24 12:56:21.904572 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:21.904207    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz" event={"ID":"3b669493-012b-4988-8f1f-3a88a4cf90ce","Type":"ContainerStarted","Data":"f2732a77c1e28bb4f50f550493925338839c9e8e51763b90e182f45d8537b420"}
Jul 24 12:56:21.904572 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:21.904264    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"
Jul 24 12:56:21.966287 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:21.966233    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz" podStartSLOduration=1.966218044 podStartE2EDuration="1.966218044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:56:21.962901686 +0000 UTC m=+2255.490648740" watchObservedRunningTime="2026-07-24 12:56:21.966218044 +0000 UTC m=+2255.493965149"
Jul 24 12:56:22.848941 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:22.848908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"
Jul 24 12:56:22.849201 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:22.849170    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"
Jul 24 12:56:23.702143 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:23.702114    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:26.910045 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:26.909997    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:56:32.911545 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:32.911513    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"
Jul 24 12:56:33.560066 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:33.560011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:33.658764 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:33.658728    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:47.287733 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.287707    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"]
Jul 24 12:56:47.288301 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.288259    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q" podUID="c1a6894c-1b1b-4c44-a0b9-58cd7b946545" containerName="maas-api" containerID="cri-o://5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2" gracePeriod=30
Jul 24 12:56:47.317694 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.317667    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:47.351981 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.351954    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"]
Jul 24 12:56:47.352224 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.352169    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2" podUID="a5317ef6-e393-4d0f-ba58-d753d09421cd" containerName="payload-processing" containerID="cri-o://7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9" gracePeriod=30
Jul 24 12:56:47.385483 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.385446    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"]
Jul 24 12:56:47.385838 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.385807    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c" podUID="bb6b407c-5f56-4c24-b070-7e0a121067a6" containerName="payload-pre-processing" containerID="cri-o://be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c" gracePeriod=30
Jul 24 12:56:47.413258 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.413228    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:47.509946 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.509796    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:47.610276 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.610240    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:47.640386 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.640359    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-openshift-service-ca\") pod \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\" (UID: \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\") "
Jul 24 12:56:47.640638 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.640620    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-maas-api-tls\") pod \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\" (UID: \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\") "
Jul 24 12:56:47.640772 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.640751    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lzg7\" (UniqueName: \"kubernetes.io/projected/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-kube-api-access-8lzg7\") pod \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\" (UID: \"c1a6894c-1b1b-4c44-a0b9-58cd7b946545\") "
Jul 24 12:56:47.640894 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.640742    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1a6894c-1b1b-4c44-a0b9-58cd7b946545" (UID: "c1a6894c-1b1b-4c44-a0b9-58cd7b946545"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:56:47.641150 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.641128    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:56:47.643712 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.643685    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1a6894c-1b1b-4c44-a0b9-58cd7b946545" (UID: "c1a6894c-1b1b-4c44-a0b9-58cd7b946545"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:56:47.643712 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.643694    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-kube-api-access-8lzg7" (OuterVolumeSpecName: "kube-api-access-8lzg7") pod "c1a6894c-1b1b-4c44-a0b9-58cd7b946545" (UID: "c1a6894c-1b1b-4c44-a0b9-58cd7b946545"). InnerVolumeSpecName "kube-api-access-8lzg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:56:47.662923 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.662898    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"
Jul 24 12:56:47.673919 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.673901    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"
Jul 24 12:56:47.741774 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.741751    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdp8d\" (UniqueName: \"kubernetes.io/projected/a5317ef6-e393-4d0f-ba58-d753d09421cd-kube-api-access-sdp8d\") pod \"a5317ef6-e393-4d0f-ba58-d753d09421cd\" (UID: \"a5317ef6-e393-4d0f-ba58-d753d09421cd\") "
Jul 24 12:56:47.741916 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.741791    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5317ef6-e393-4d0f-ba58-d753d09421cd-plugins-config-volume\") pod \"a5317ef6-e393-4d0f-ba58-d753d09421cd\" (UID: \"a5317ef6-e393-4d0f-ba58-d753d09421cd\") "
Jul 24 12:56:47.741916 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.741827    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzv4s\" (UniqueName: \"kubernetes.io/projected/bb6b407c-5f56-4c24-b070-7e0a121067a6-kube-api-access-fzv4s\") pod \"bb6b407c-5f56-4c24-b070-7e0a121067a6\" (UID: \"bb6b407c-5f56-4c24-b070-7e0a121067a6\") "
Jul 24 12:56:47.741916 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.741858    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6b407c-5f56-4c24-b070-7e0a121067a6-plugins-config-volume\") pod \"bb6b407c-5f56-4c24-b070-7e0a121067a6\" (UID: \"bb6b407c-5f56-4c24-b070-7e0a121067a6\") "
Jul 24 12:56:47.742088 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.742075    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:56:47.742149 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.742092    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lzg7\" (UniqueName: \"kubernetes.io/projected/c1a6894c-1b1b-4c44-a0b9-58cd7b946545-kube-api-access-8lzg7\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:56:47.742273 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.742249    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5317ef6-e393-4d0f-ba58-d753d09421cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5317ef6-e393-4d0f-ba58-d753d09421cd" (UID: "a5317ef6-e393-4d0f-ba58-d753d09421cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:56:47.742371 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.742350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6b407c-5f56-4c24-b070-7e0a121067a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb6b407c-5f56-4c24-b070-7e0a121067a6" (UID: "bb6b407c-5f56-4c24-b070-7e0a121067a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:56:47.743988 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.743972    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6b407c-5f56-4c24-b070-7e0a121067a6-kube-api-access-fzv4s" (OuterVolumeSpecName: "kube-api-access-fzv4s") pod "bb6b407c-5f56-4c24-b070-7e0a121067a6" (UID: "bb6b407c-5f56-4c24-b070-7e0a121067a6"). InnerVolumeSpecName "kube-api-access-fzv4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:56:47.744064 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.744009    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5317ef6-e393-4d0f-ba58-d753d09421cd-kube-api-access-sdp8d" (OuterVolumeSpecName: "kube-api-access-sdp8d") pod "a5317ef6-e393-4d0f-ba58-d753d09421cd" (UID: "a5317ef6-e393-4d0f-ba58-d753d09421cd"). InnerVolumeSpecName "kube-api-access-sdp8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:56:47.842596 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.842541    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdp8d\" (UniqueName: \"kubernetes.io/projected/a5317ef6-e393-4d0f-ba58-d753d09421cd-kube-api-access-sdp8d\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:56:47.842596 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.842562    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5317ef6-e393-4d0f-ba58-d753d09421cd-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:56:47.842596 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.842571    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzv4s\" (UniqueName: \"kubernetes.io/projected/bb6b407c-5f56-4c24-b070-7e0a121067a6-kube-api-access-fzv4s\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:56:47.842596 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:47.842581    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6b407c-5f56-4c24-b070-7e0a121067a6-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:56:48.027088 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.027062    2566 generic.go:356] "Generic (PLEG): container finished" podID="c1a6894c-1b1b-4c44-a0b9-58cd7b946545" containerID="5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2" exitCode=0
Jul 24 12:56:48.027197 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.027118    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"
Jul 24 12:56:48.027197 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.027146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q" event={"ID":"c1a6894c-1b1b-4c44-a0b9-58cd7b946545","Type":"ContainerDied","Data":"5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2"}
Jul 24 12:56:48.027197 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.027192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q" event={"ID":"c1a6894c-1b1b-4c44-a0b9-58cd7b946545","Type":"ContainerDied","Data":"f111eef4b4ff63a098d9f31dcc1d749df2b63d94d3fffdf8a70d204f4dc77732"}
Jul 24 12:56:48.027354 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.027225    2566 scope.go:117] "RemoveContainer" containerID="5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2"
Jul 24 12:56:48.028600 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.028577    2566 generic.go:356] "Generic (PLEG): container finished" podID="bb6b407c-5f56-4c24-b070-7e0a121067a6" containerID="be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c" exitCode=0
Jul 24 12:56:48.028684 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.028616    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"
Jul 24 12:56:48.028684 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.028661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c" event={"ID":"bb6b407c-5f56-4c24-b070-7e0a121067a6","Type":"ContainerDied","Data":"be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c"}
Jul 24 12:56:48.028781 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.028699    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c" event={"ID":"bb6b407c-5f56-4c24-b070-7e0a121067a6","Type":"ContainerDied","Data":"67a2656a469f303c5884a80b3ef6eb241dbc2ef7c76a66c3bfb8f5a7b64c829c"}
Jul 24 12:56:48.030090 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.030066    2566 generic.go:356] "Generic (PLEG): container finished" podID="a5317ef6-e393-4d0f-ba58-d753d09421cd" containerID="7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9" exitCode=0
Jul 24 12:56:48.030208 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.030094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2" event={"ID":"a5317ef6-e393-4d0f-ba58-d753d09421cd","Type":"ContainerDied","Data":"7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9"}
Jul 24 12:56:48.030208 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.030145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2" event={"ID":"a5317ef6-e393-4d0f-ba58-d753d09421cd","Type":"ContainerDied","Data":"32dfda05016a9bc94f92278993fc32bd9aaafad6a31d48044182456dd747bd1b"}
Jul 24 12:56:48.030315 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.030298    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"
Jul 24 12:56:48.037992 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.037978    2566 scope.go:117] "RemoveContainer" containerID="5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2"
Jul 24 12:56:48.038271 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:56:48.038251    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2\": container with ID starting with 5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2 not found: ID does not exist" containerID="5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2"
Jul 24 12:56:48.038355 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.038282    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2"} err="failed to get container status \"5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2\": rpc error: code = NotFound desc = could not find container \"5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2\": container with ID starting with 5d7e5ae778ae72027d8bec4157900569a3f65cb907758d1ba7e9bfedf7e94aa2 not found: ID does not exist"
Jul 24 12:56:48.038355 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.038303    2566 scope.go:117] "RemoveContainer" containerID="be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c"
Jul 24 12:56:48.046266 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.046252    2566 scope.go:117] "RemoveContainer" containerID="be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c"
Jul 24 12:56:48.046485 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:56:48.046468    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c\": container with ID starting with be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c not found: ID does not exist" containerID="be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c"
Jul 24 12:56:48.046556 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.046496    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c"} err="failed to get container status \"be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c\": rpc error: code = NotFound desc = could not find container \"be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c\": container with ID starting with be821b2f773c426723f726cf0e17d821fdb1cc793032200c6f5547c27ebb3c1c not found: ID does not exist"
Jul 24 12:56:48.046556 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.046532    2566 scope.go:117] "RemoveContainer" containerID="7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9"
Jul 24 12:56:48.053785 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.053764    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"]
Jul 24 12:56:48.054425 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.054412    2566 scope.go:117] "RemoveContainer" containerID="7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9"
Jul 24 12:56:48.054650 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:56:48.054633    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9\": container with ID starting with 7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9 not found: ID does not exist" containerID="7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9"
Jul 24 12:56:48.054709 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.054659    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9"} err="failed to get container status \"7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9\": rpc error: code = NotFound desc = could not find container \"7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9\": container with ID starting with 7878634fbed767ac9312ca15eb16531faccdad8b89392915b320bcc3b27764b9 not found: ID does not exist"
Jul 24 12:56:48.058266 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.058236    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f394b5-5bfb7f7457-vzc7q"]
Jul 24 12:56:48.070183 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.070162    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"]
Jul 24 12:56:48.075599 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.075581    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f394b5-786777cc4b-wxkh2"]
Jul 24 12:56:48.086548 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.086529    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"]
Jul 24 12:56:48.090649 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.090630    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f394b5-7c4bf579cb-z8d7c"]
Jul 24 12:56:48.974671 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.974642    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5317ef6-e393-4d0f-ba58-d753d09421cd" path="/var/lib/kubelet/pods/a5317ef6-e393-4d0f-ba58-d753d09421cd/volumes"
Jul 24 12:56:48.975054 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.974997    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb6b407c-5f56-4c24-b070-7e0a121067a6" path="/var/lib/kubelet/pods/bb6b407c-5f56-4c24-b070-7e0a121067a6/volumes"
Jul 24 12:56:48.975395 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:48.975379    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a6894c-1b1b-4c44-a0b9-58cd7b946545" path="/var/lib/kubelet/pods/c1a6894c-1b1b-4c44-a0b9-58cd7b946545/volumes"
Jul 24 12:56:52.027321 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:52.027290    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:52.435166 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:52.435081    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:58.675957 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:58.675924    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:58.776470 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:58.776428    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:58.981341 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:58.981250    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:56:59.235151 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:56:59.235066    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:14.965734 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:14.965172    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"]
Jul 24 12:57:14.965734 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:14.965390    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz" podUID="eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11" containerName="maas-api" containerID="cri-o://1d2e12be94ba5b1e64e2aecb1b5991cf105b6785d0e508c5293ef680626c0e75" gracePeriod=30
Jul 24 12:57:14.998144 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:14.998118    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:15.033953 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.033925    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"]
Jul 24 12:57:15.034198 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.034166    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz" podUID="3b669493-012b-4988-8f1f-3a88a4cf90ce" containerName="payload-processing" containerID="cri-o://6e3cee67cb964cb5a871b197a32ad7881fd0a4e0bd88e1574cc1102eb343af87" gracePeriod=30
Jul 24 12:57:15.076797 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.076771    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:15.160770 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.160734    2566 generic.go:356] "Generic (PLEG): container finished" podID="eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11" containerID="1d2e12be94ba5b1e64e2aecb1b5991cf105b6785d0e508c5293ef680626c0e75" exitCode=0
Jul 24 12:57:15.160931 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.160771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz" event={"ID":"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11","Type":"ContainerDied","Data":"1d2e12be94ba5b1e64e2aecb1b5991cf105b6785d0e508c5293ef680626c0e75"}
Jul 24 12:57:15.162676 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.162649    2566 generic.go:356] "Generic (PLEG): container finished" podID="3b669493-012b-4988-8f1f-3a88a4cf90ce" containerID="6e3cee67cb964cb5a871b197a32ad7881fd0a4e0bd88e1574cc1102eb343af87" exitCode=0
Jul 24 12:57:15.162797 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.162693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz" event={"ID":"3b669493-012b-4988-8f1f-3a88a4cf90ce","Type":"ContainerDied","Data":"6e3cee67cb964cb5a871b197a32ad7881fd0a4e0bd88e1574cc1102eb343af87"}
Jul 24 12:57:15.178812 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.178757    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:15.287791 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.287767    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:57:15.308120 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.308100    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"
Jul 24 12:57:15.355902 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.355880    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b669493-012b-4988-8f1f-3a88a4cf90ce-plugins-config-volume\") pod \"3b669493-012b-4988-8f1f-3a88a4cf90ce\" (UID: \"3b669493-012b-4988-8f1f-3a88a4cf90ce\") "
Jul 24 12:57:15.356071 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.355921    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-openshift-service-ca\") pod \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\" (UID: \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\") "
Jul 24 12:57:15.356071 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.355940    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wf5x\" (UniqueName: \"kubernetes.io/projected/3b669493-012b-4988-8f1f-3a88a4cf90ce-kube-api-access-9wf5x\") pod \"3b669493-012b-4988-8f1f-3a88a4cf90ce\" (UID: \"3b669493-012b-4988-8f1f-3a88a4cf90ce\") "
Jul 24 12:57:15.356071 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.355978    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-maas-api-tls\") pod \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\" (UID: \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\") "
Jul 24 12:57:15.356071 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.356046    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgrtf\" (UniqueName: \"kubernetes.io/projected/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-kube-api-access-qgrtf\") pod \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\" (UID: \"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11\") "
Jul 24 12:57:15.356336 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.356309    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11" (UID: "eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:57:15.356420 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.356332    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b669493-012b-4988-8f1f-3a88a4cf90ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b669493-012b-4988-8f1f-3a88a4cf90ce" (UID: "3b669493-012b-4988-8f1f-3a88a4cf90ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:57:15.358183 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.358155    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b669493-012b-4988-8f1f-3a88a4cf90ce-kube-api-access-9wf5x" (OuterVolumeSpecName: "kube-api-access-9wf5x") pod "3b669493-012b-4988-8f1f-3a88a4cf90ce" (UID: "3b669493-012b-4988-8f1f-3a88a4cf90ce"). InnerVolumeSpecName "kube-api-access-9wf5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:57:15.358285 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.358246    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-kube-api-access-qgrtf" (OuterVolumeSpecName: "kube-api-access-qgrtf") pod "eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11" (UID: "eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11"). InnerVolumeSpecName "kube-api-access-qgrtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:57:15.358558 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.358536    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11" (UID: "eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:57:15.457293 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.457272    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgrtf\" (UniqueName: \"kubernetes.io/projected/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-kube-api-access-qgrtf\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:57:15.457293 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.457292    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b669493-012b-4988-8f1f-3a88a4cf90ce-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:57:15.457429 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.457300    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:57:15.457429 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.457309    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wf5x\" (UniqueName: \"kubernetes.io/projected/3b669493-012b-4988-8f1f-3a88a4cf90ce-kube-api-access-9wf5x\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:57:15.457429 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:15.457320    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 12:57:16.168708 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:16.168675    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"
Jul 24 12:57:16.168708 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:16.168700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz" event={"ID":"3b669493-012b-4988-8f1f-3a88a4cf90ce","Type":"ContainerDied","Data":"f2732a77c1e28bb4f50f550493925338839c9e8e51763b90e182f45d8537b420"}
Jul 24 12:57:16.169174 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:16.168756    2566 scope.go:117] "RemoveContainer" containerID="6e3cee67cb964cb5a871b197a32ad7881fd0a4e0bd88e1574cc1102eb343af87"
Jul 24 12:57:16.170313 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:16.170294    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"
Jul 24 12:57:16.170313 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:16.170303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz" event={"ID":"eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11","Type":"ContainerDied","Data":"57c5f262dc5a39bcebc1aec0c57be6a550d85f1634ec81e89f965a79a82c5ae3"}
Jul 24 12:57:16.178847 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:16.178832    2566 scope.go:117] "RemoveContainer" containerID="1d2e12be94ba5b1e64e2aecb1b5991cf105b6785d0e508c5293ef680626c0e75"
Jul 24 12:57:16.194120 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:16.194098    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"]
Jul 24 12:57:16.197591 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:16.197571    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aa0820-7984d87586-rv4hz"]
Jul 24 12:57:16.209625 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:16.209551    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"]
Jul 24 12:57:16.211580 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:16.211564    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aa0820-f947b88c7-bz4kz"]
Jul 24 12:57:16.974958 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:16.974927    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b669493-012b-4988-8f1f-3a88a4cf90ce" path="/var/lib/kubelet/pods/3b669493-012b-4988-8f1f-3a88a4cf90ce/volumes"
Jul 24 12:57:16.975306 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:16.975292    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11" path="/var/lib/kubelet/pods/eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11/volumes"
Jul 24 12:57:19.450853 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:19.450814    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:19.890216 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:19.890178    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:26.021438 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:26.021401    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:26.121216 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:26.121189    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:26.329136 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:26.329044    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:26.579000 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:26.578969    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:26.961487 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:26.961451    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:27.308491 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:27.308461    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:28.043611 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:28.043572    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:30.138465 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.138429    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:30.233379 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.233338    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:30.242263 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242238    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"]
Jul 24 12:57:30.242741 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242723    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5317ef6-e393-4d0f-ba58-d753d09421cd" containerName="payload-processing"
Jul 24 12:57:30.242741 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242744    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5317ef6-e393-4d0f-ba58-d753d09421cd" containerName="payload-processing"
Jul 24 12:57:30.242912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242767    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a6894c-1b1b-4c44-a0b9-58cd7b946545" containerName="maas-api"
Jul 24 12:57:30.242912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242776    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a6894c-1b1b-4c44-a0b9-58cd7b946545" containerName="maas-api"
Jul 24 12:57:30.242912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242799    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb6b407c-5f56-4c24-b070-7e0a121067a6" containerName="payload-pre-processing"
Jul 24 12:57:30.242912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242807    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb6b407c-5f56-4c24-b070-7e0a121067a6" containerName="payload-pre-processing"
Jul 24 12:57:30.242912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242823    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b669493-012b-4988-8f1f-3a88a4cf90ce" containerName="payload-processing"
Jul 24 12:57:30.242912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242831    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b669493-012b-4988-8f1f-3a88a4cf90ce" containerName="payload-processing"
Jul 24 12:57:30.242912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242845    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11" containerName="maas-api"
Jul 24 12:57:30.242912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242853    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11" containerName="maas-api"
Jul 24 12:57:30.243344 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242980    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaeeee5f-55ae-4903-8b4a-8e49a1a2eb11" containerName="maas-api"
Jul 24 12:57:30.243344 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.242995    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b669493-012b-4988-8f1f-3a88a4cf90ce" containerName="payload-processing"
Jul 24 12:57:30.243344 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.243008    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5317ef6-e393-4d0f-ba58-d753d09421cd" containerName="payload-processing"
Jul 24 12:57:30.243344 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.243043    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a6894c-1b1b-4c44-a0b9-58cd7b946545" containerName="maas-api"
Jul 24 12:57:30.243344 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.243057    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb6b407c-5f56-4c24-b070-7e0a121067a6" containerName="payload-pre-processing"
Jul 24 12:57:30.246135 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.246117    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:30.249579 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.249563    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-d15f64-serving-cert\""
Jul 24 12:57:30.262226 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.262203    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"]
Jul 24 12:57:30.309757 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.309729    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:30.382252 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.382229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-maas-api-tls\") pod \"maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9\" (UID: \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:30.382372 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.382279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4dt8\" (UniqueName: \"kubernetes.io/projected/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-kube-api-access-n4dt8\") pod \"maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9\" (UID: \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:30.382372 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.382327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9\" (UID: \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:30.483755 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.483693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4dt8\" (UniqueName: \"kubernetes.io/projected/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-kube-api-access-n4dt8\") pod \"maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9\" (UID: \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:30.483755 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.483723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9\" (UID: \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:30.483893 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.483777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-maas-api-tls\") pod \"maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9\" (UID: \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:30.484472 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.484449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9\" (UID: \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:30.486207 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.486187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-maas-api-tls\") pod \"maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9\" (UID: \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:30.492516 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.492493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4dt8\" (UniqueName: \"kubernetes.io/projected/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-kube-api-access-n4dt8\") pod \"maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9\" (UID: \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:30.509821 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.509782    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:30.556685 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.556667    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:30.904248 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.904216    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"]
Jul 24 12:57:30.905652 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:57:30.905627    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9a0f38_a4e3_4967_8894_fc5d84c99e54.slice/crio-1fd8020cd87086cdbac7197fc0a10223a7271b2898394b48bde8ac8f37605e19 WatchSource:0}: Error finding container 1fd8020cd87086cdbac7197fc0a10223a7271b2898394b48bde8ac8f37605e19: Status 404 returned error can't find the container with id 1fd8020cd87086cdbac7197fc0a10223a7271b2898394b48bde8ac8f37605e19
Jul 24 12:57:30.994916 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:30.994883    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:31.092974 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:31.092941    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:31.241420 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:31.241346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9" event={"ID":"ad9a0f38-a4e3-4967-8894-fc5d84c99e54","Type":"ContainerStarted","Data":"1fd8020cd87086cdbac7197fc0a10223a7271b2898394b48bde8ac8f37605e19"}
Jul 24 12:57:32.248070 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:32.248008    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9" event={"ID":"ad9a0f38-a4e3-4967-8894-fc5d84c99e54","Type":"ContainerStarted","Data":"ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364"}
Jul 24 12:57:32.248472 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:32.248087    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:32.270912 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:32.270868    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9" podStartSLOduration=1.767164822 podStartE2EDuration="2.270854056s" totalImagesPullingTime="503.689234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:57:32.269625526 +0000 UTC m=+2325.797372580" watchObservedRunningTime="2026-07-24 12:57:32.270854056 +0000 UTC m=+2325.798601109"
Jul 24 12:57:34.702657 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:34.702627    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:35.107000 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:35.106967    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:35.418462 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:35.418373    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:35.667997 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:35.667954    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:38.258557 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.258487    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 12:57:38.277114 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.277088    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:38.386557 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.386525    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"]
Jul 24 12:57:38.390122 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.390103    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:38.394231 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.394212    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c6697a-serving-cert\""
Jul 24 12:57:38.411644 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.411617    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"]
Jul 24 12:57:38.468292 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.468267    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:38.550505 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.550437    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:38.550505 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.550491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:38.550942 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.550538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz8ts\" (UniqueName: \"kubernetes.io/projected/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-kube-api-access-rz8ts\") pod \"maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:38.565360 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.565340    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:38.651431 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.651407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:38.651520 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.651460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:38.651586 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.651517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8ts\" (UniqueName: \"kubernetes.io/projected/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-kube-api-access-rz8ts\") pod \"maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:38.651643 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:57:38.651610    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-serving-cert: secret "maas-api-e2e-shared-b-c6697a-serving-cert" not found
Jul 24 12:57:38.651701 ip-10-0-140-61 kubenswrapper[2566]: E0724 12:57:38.651688    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-maas-api-tls podName:7955473c-fa61-4fc1-ac43-ff38e6bf28e9 nodeName:}" failed. No retries permitted until 2026-07-24 12:57:39.151666171 +0000 UTC m=+2332.679413209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-maas-api-tls") pod "maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv" (UID: "7955473c-fa61-4fc1-ac43-ff38e6bf28e9") : secret "maas-api-e2e-shared-b-c6697a-serving-cert" not found
Jul 24 12:57:38.651996 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.651979    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:38.661045 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.661007    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8ts\" (UniqueName: \"kubernetes.io/projected/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-kube-api-access-rz8ts\") pod \"maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:38.726942 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:38.726908    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:39.036526 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:39.036486    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:39.138241 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:39.138208    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:39.156322 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:39.156297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:39.159243 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:39.159219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:39.303592 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:39.303517    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:39.441948 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:39.441915    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"]
Jul 24 12:57:39.442949 ip-10-0-140-61 kubenswrapper[2566]: W0724 12:57:39.442919    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7955473c_fa61_4fc1_ac43_ff38e6bf28e9.slice/crio-7c11c4c639710d15d8a721478685d1988837fd5778db80cc08e47fd610d9e473 WatchSource:0}: Error finding container 7c11c4c639710d15d8a721478685d1988837fd5778db80cc08e47fd610d9e473: Status 404 returned error can't find the container with id 7c11c4c639710d15d8a721478685d1988837fd5778db80cc08e47fd610d9e473
Jul 24 12:57:40.287986 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:40.287953    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv" event={"ID":"7955473c-fa61-4fc1-ac43-ff38e6bf28e9","Type":"ContainerStarted","Data":"6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d"}
Jul 24 12:57:40.288163 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:40.288031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv" event={"ID":"7955473c-fa61-4fc1-ac43-ff38e6bf28e9","Type":"ContainerStarted","Data":"7c11c4c639710d15d8a721478685d1988837fd5778db80cc08e47fd610d9e473"}
Jul 24 12:57:40.288163 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:40.288066    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:57:40.307185 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:40.307141    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv" podStartSLOduration=1.8877656539999998 podStartE2EDuration="2.307127494s" totalImagesPullingTime="419.36184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:57:40.305312214 +0000 UTC m=+2333.833059267" watchObservedRunningTime="2026-07-24 12:57:40.307127494 +0000 UTC m=+2333.834874575"
Jul 24 12:57:42.772445 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:42.772417    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:57:46.298277 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:57:46.298227    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 12:58:03.156435 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:58:03.156397    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:58:10.690991 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:58:10.690955    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:58:10.792773 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:58:10.792745    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:58:26.776287 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:58:26.776249    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 12:58:47.141443 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:58:47.141407    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 12:58:47.142866 ip-10-0-140-61 kubenswrapper[2566]: I0724 12:58:47.142844    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 13:00:33.257435 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:00:33.257401    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:00:34.450878 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:00:34.450835    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:01:02.077277 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:01:02.077244    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:01:20.765482 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:01:20.765446    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:01:42.163466 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:01:42.163417    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:01:56.316683 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:01:56.316642    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:01:57.484571 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:01:57.484522    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:02:14.677235 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:02:14.677198    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:02:31.882923 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:02:31.882882    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:26.067899 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:26.067858    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:31.533182 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:31.533146    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:32.324076 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:32.324040    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:47.179985 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:47.179908    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 13:03:47.182088 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:47.182065    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 13:03:52.443740 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:52.443710    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:52.808928 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:52.808881    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:53.043560 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:53.043514    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:56.308595 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.308559    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:56.405310 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.405263    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:56.416961 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.416924    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"]
Jul 24 13:03:56.420422 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.420402    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:03:56.423496 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.423478    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-967d3e-serving-cert\""
Jul 24 13:03:56.431781 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.431746    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"]
Jul 24 13:03:56.504106 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.504050    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:56.526862 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.526828    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34229da8-f49e-4b19-b1e5-f8ffddb56e02-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt\" (UID: \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:03:56.527068 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.526893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6mdb\" (UniqueName: \"kubernetes.io/projected/34229da8-f49e-4b19-b1e5-f8ffddb56e02-kube-api-access-m6mdb\") pod \"maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt\" (UID: \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:03:56.527068 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.526964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34229da8-f49e-4b19-b1e5-f8ffddb56e02-maas-api-tls\") pod \"maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt\" (UID: \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:03:56.628181 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.628096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34229da8-f49e-4b19-b1e5-f8ffddb56e02-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt\" (UID: \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:03:56.628181 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.628157    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6mdb\" (UniqueName: \"kubernetes.io/projected/34229da8-f49e-4b19-b1e5-f8ffddb56e02-kube-api-access-m6mdb\") pod \"maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt\" (UID: \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:03:56.628392 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.628239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34229da8-f49e-4b19-b1e5-f8ffddb56e02-maas-api-tls\") pod \"maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt\" (UID: \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:03:56.628814 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.628786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34229da8-f49e-4b19-b1e5-f8ffddb56e02-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt\" (UID: \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:03:56.630812 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.630785    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34229da8-f49e-4b19-b1e5-f8ffddb56e02-maas-api-tls\") pod \"maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt\" (UID: \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:03:56.634174 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.634152    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:56.637718 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.637696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6mdb\" (UniqueName: \"kubernetes.io/projected/34229da8-f49e-4b19-b1e5-f8ffddb56e02-kube-api-access-m6mdb\") pod \"maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt\" (UID: \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:03:56.730746 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.730710    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:03:56.867593 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.867565    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"]
Jul 24 13:03:56.868546 ip-10-0-140-61 kubenswrapper[2566]: W0724 13:03:56.868521    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34229da8_f49e_4b19_b1e5_f8ffddb56e02.slice/crio-2becc707f041f1dab567a0fb954746f250ee06f355e0c5da43f3dd105350b787 WatchSource:0}: Error finding container 2becc707f041f1dab567a0fb954746f250ee06f355e0c5da43f3dd105350b787: Status 404 returned error can't find the container with id 2becc707f041f1dab567a0fb954746f250ee06f355e0c5da43f3dd105350b787
Jul 24 13:03:56.869943 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:56.869921    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:03:57.123339 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.123310    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:57.125978 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.125948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt" event={"ID":"34229da8-f49e-4b19-b1e5-f8ffddb56e02","Type":"ContainerStarted","Data":"2becc707f041f1dab567a0fb954746f250ee06f355e0c5da43f3dd105350b787"}
Jul 24 13:03:57.225879 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.225843    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:03:57.253094 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.253054    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"]
Jul 24 13:03:57.258221 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.258198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"
Jul 24 13:03:57.261364 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.261336    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-967d3e-dockercfg-wrgrq\""
Jul 24 13:03:57.262727 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.262701    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-967d3e\""
Jul 24 13:03:57.268558 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.268536    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"]
Jul 24 13:03:57.435908 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.435819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fklq2\" (UniqueName: \"kubernetes.io/projected/19172175-1a85-4f9d-9565-fa066472143c-kube-api-access-fklq2\") pod \"payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln\" (UID: \"19172175-1a85-4f9d-9565-fa066472143c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"
Jul 24 13:03:57.435908 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.435873    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19172175-1a85-4f9d-9565-fa066472143c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln\" (UID: \"19172175-1a85-4f9d-9565-fa066472143c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"
Jul 24 13:03:57.537275 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.537243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fklq2\" (UniqueName: \"kubernetes.io/projected/19172175-1a85-4f9d-9565-fa066472143c-kube-api-access-fklq2\") pod \"payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln\" (UID: \"19172175-1a85-4f9d-9565-fa066472143c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"
Jul 24 13:03:57.537456 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.537314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19172175-1a85-4f9d-9565-fa066472143c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln\" (UID: \"19172175-1a85-4f9d-9565-fa066472143c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"
Jul 24 13:03:57.537975 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.537957    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19172175-1a85-4f9d-9565-fa066472143c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln\" (UID: \"19172175-1a85-4f9d-9565-fa066472143c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"
Jul 24 13:03:57.546188 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.546159    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fklq2\" (UniqueName: \"kubernetes.io/projected/19172175-1a85-4f9d-9565-fa066472143c-kube-api-access-fklq2\") pod \"payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln\" (UID: \"19172175-1a85-4f9d-9565-fa066472143c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"
Jul 24 13:03:57.571446 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.571418    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"
Jul 24 13:03:57.723224 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:57.723199    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"]
Jul 24 13:03:57.725664 ip-10-0-140-61 kubenswrapper[2566]: W0724 13:03:57.725634    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19172175_1a85_4f9d_9565_fa066472143c.slice/crio-1e1d4610d41109b69e379acf4f7216dafb742b645d64582111ca7aff51228f0e WatchSource:0}: Error finding container 1e1d4610d41109b69e379acf4f7216dafb742b645d64582111ca7aff51228f0e: Status 404 returned error can't find the container with id 1e1d4610d41109b69e379acf4f7216dafb742b645d64582111ca7aff51228f0e
Jul 24 13:03:58.131935 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:58.131900    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln" event={"ID":"19172175-1a85-4f9d-9565-fa066472143c","Type":"ContainerStarted","Data":"a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0"}
Jul 24 13:03:58.132148 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:58.131983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln" event={"ID":"19172175-1a85-4f9d-9565-fa066472143c","Type":"ContainerStarted","Data":"1e1d4610d41109b69e379acf4f7216dafb742b645d64582111ca7aff51228f0e"}
Jul 24 13:03:58.132148 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:58.132040    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"
Jul 24 13:03:58.133577 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:58.133551    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt" event={"ID":"34229da8-f49e-4b19-b1e5-f8ffddb56e02","Type":"ContainerStarted","Data":"914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587"}
Jul 24 13:03:58.133697 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:58.133608    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:03:58.151672 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:58.151613    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln" podStartSLOduration=1.151599237 podStartE2EDuration="1.151599237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:03:58.148424545 +0000 UTC m=+2711.676171597" watchObservedRunningTime="2026-07-24 13:03:58.151599237 +0000 UTC m=+2711.679346289"
Jul 24 13:03:58.168475 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:03:58.168425    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt" podStartSLOduration=1.473992564 podStartE2EDuration="2.168410638s" totalImagesPullingTime="694.418074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:03:58.165611507 +0000 UTC m=+2711.693358561" watchObservedRunningTime="2026-07-24 13:03:58.168410638 +0000 UTC m=+2711.696157691"
Jul 24 13:04:01.036789 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:01.036754    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:04:03.180527 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:03.180500    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:04:04.144778 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:04.144746    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:04:09.141344 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:09.141314    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"
Jul 24 13:04:11.777430 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:11.777397    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:04:12.101783 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:12.101688    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:04:12.371174 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:12.371052    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:04:15.452991 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.452957    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:04:15.504231 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.504202    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"]
Jul 24 13:04:15.508879 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.508859    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:15.512299 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.512276    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-47a7ae-serving-cert\""
Jul 24 13:04:15.519289 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.519258    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"]
Jul 24 13:04:15.545140 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.545103    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:04:15.608296 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.608269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99862a98-6e57-459d-a963-6494863dcf51-maas-api-tls\") pod \"maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:15.608457 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.608304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99862a98-6e57-459d-a963-6494863dcf51-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:15.608457 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.608395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8js6\" (UniqueName: \"kubernetes.io/projected/99862a98-6e57-459d-a963-6494863dcf51-kube-api-access-f8js6\") pod \"maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:15.618674 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.618645    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:04:15.709171 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.709083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99862a98-6e57-459d-a963-6494863dcf51-maas-api-tls\") pod \"maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:15.709171 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.709124    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99862a98-6e57-459d-a963-6494863dcf51-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:15.709388 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.709193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8js6\" (UniqueName: \"kubernetes.io/projected/99862a98-6e57-459d-a963-6494863dcf51-kube-api-access-f8js6\") pod \"maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:15.709388 ip-10-0-140-61 kubenswrapper[2566]: E0724 13:04:15.709261    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-serving-cert: secret "maas-api-e2e-ipp-b-47a7ae-serving-cert" not found
Jul 24 13:04:15.709388 ip-10-0-140-61 kubenswrapper[2566]: E0724 13:04:15.709340    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99862a98-6e57-459d-a963-6494863dcf51-maas-api-tls podName:99862a98-6e57-459d-a963-6494863dcf51 nodeName:}" failed. No retries permitted until 2026-07-24 13:04:16.209319975 +0000 UTC m=+2729.737067010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99862a98-6e57-459d-a963-6494863dcf51-maas-api-tls") pod "maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f" (UID: "99862a98-6e57-459d-a963-6494863dcf51") : secret "maas-api-e2e-ipp-b-47a7ae-serving-cert" not found
Jul 24 13:04:15.709897 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.709879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99862a98-6e57-459d-a963-6494863dcf51-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:15.719468 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.719447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8js6\" (UniqueName: \"kubernetes.io/projected/99862a98-6e57-459d-a963-6494863dcf51-kube-api-access-f8js6\") pod \"maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:15.899168 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:15.899136    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:04:16.214296 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.214268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99862a98-6e57-459d-a963-6494863dcf51-maas-api-tls\") pod \"maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:16.216757 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.216730    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99862a98-6e57-459d-a963-6494863dcf51-maas-api-tls\") pod \"maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:16.369246 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.369217    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:04:16.420545 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.420515    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:16.469089 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.468980    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:04:16.483338 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.483309    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"]
Jul 24 13:04:16.487940 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.487911    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"
Jul 24 13:04:16.494329 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.494292    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-47a7ae\""
Jul 24 13:04:16.496884 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.496848    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-47a7ae-dockercfg-wkhbg\""
Jul 24 13:04:16.500795 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.500769    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"]
Jul 24 13:04:16.517487 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.517458    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a49d324-04b6-4099-b988-a1663eaeb6a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7\" (UID: \"5a49d324-04b6-4099-b988-a1663eaeb6a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"
Jul 24 13:04:16.517589 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.517557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqscx\" (UniqueName: \"kubernetes.io/projected/5a49d324-04b6-4099-b988-a1663eaeb6a5-kube-api-access-dqscx\") pod \"payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7\" (UID: \"5a49d324-04b6-4099-b988-a1663eaeb6a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"
Jul 24 13:04:16.571916 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.571885    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"]
Jul 24 13:04:16.572764 ip-10-0-140-61 kubenswrapper[2566]: W0724 13:04:16.572740    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99862a98_6e57_459d_a963_6494863dcf51.slice/crio-3ce4b194c6b8a6c614cdc5a73ac6f2beb4b30f5ce30819c4478370b721e65ea4 WatchSource:0}: Error finding container 3ce4b194c6b8a6c614cdc5a73ac6f2beb4b30f5ce30819c4478370b721e65ea4: Status 404 returned error can't find the container with id 3ce4b194c6b8a6c614cdc5a73ac6f2beb4b30f5ce30819c4478370b721e65ea4
Jul 24 13:04:16.618431 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.618405    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a49d324-04b6-4099-b988-a1663eaeb6a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7\" (UID: \"5a49d324-04b6-4099-b988-a1663eaeb6a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"
Jul 24 13:04:16.618567 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.618483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqscx\" (UniqueName: \"kubernetes.io/projected/5a49d324-04b6-4099-b988-a1663eaeb6a5-kube-api-access-dqscx\") pod \"payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7\" (UID: \"5a49d324-04b6-4099-b988-a1663eaeb6a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"
Jul 24 13:04:16.619101 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.619084    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a49d324-04b6-4099-b988-a1663eaeb6a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7\" (UID: \"5a49d324-04b6-4099-b988-a1663eaeb6a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"
Jul 24 13:04:16.628421 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.628391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqscx\" (UniqueName: \"kubernetes.io/projected/5a49d324-04b6-4099-b988-a1663eaeb6a5-kube-api-access-dqscx\") pod \"payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7\" (UID: \"5a49d324-04b6-4099-b988-a1663eaeb6a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"
Jul 24 13:04:16.805586 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.805559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"
Jul 24 13:04:16.943360 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:16.943329    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"]
Jul 24 13:04:16.945289 ip-10-0-140-61 kubenswrapper[2566]: W0724 13:04:16.945266    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a49d324_04b6_4099_b988_a1663eaeb6a5.slice/crio-c8e5747e0efe19ed14cd9e806fe37b68825a647e1d759a20c2b53139c68b1461 WatchSource:0}: Error finding container c8e5747e0efe19ed14cd9e806fe37b68825a647e1d759a20c2b53139c68b1461: Status 404 returned error can't find the container with id c8e5747e0efe19ed14cd9e806fe37b68825a647e1d759a20c2b53139c68b1461
Jul 24 13:04:17.231593 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:17.231493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7" event={"ID":"5a49d324-04b6-4099-b988-a1663eaeb6a5","Type":"ContainerStarted","Data":"07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8"}
Jul 24 13:04:17.231593 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:17.231555    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7" event={"ID":"5a49d324-04b6-4099-b988-a1663eaeb6a5","Type":"ContainerStarted","Data":"c8e5747e0efe19ed14cd9e806fe37b68825a647e1d759a20c2b53139c68b1461"}
Jul 24 13:04:17.231593 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:17.231586    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"
Jul 24 13:04:17.233000 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:17.232979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f" event={"ID":"99862a98-6e57-459d-a963-6494863dcf51","Type":"ContainerStarted","Data":"25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422"}
Jul 24 13:04:17.233089 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:17.233043    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f" event={"ID":"99862a98-6e57-459d-a963-6494863dcf51","Type":"ContainerStarted","Data":"3ce4b194c6b8a6c614cdc5a73ac6f2beb4b30f5ce30819c4478370b721e65ea4"}
Jul 24 13:04:17.233131 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:17.233091    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:17.254849 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:17.254808    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7" podStartSLOduration=1.254796107 podStartE2EDuration="1.254796107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:04:17.25290532 +0000 UTC m=+2730.780652375" watchObservedRunningTime="2026-07-24 13:04:17.254796107 +0000 UTC m=+2730.782543139"
Jul 24 13:04:17.274876 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:17.274831    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f" podStartSLOduration=1.9103467589999998 podStartE2EDuration="2.274817808s" totalImagesPullingTime="364.471049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:04:17.27332576 +0000 UTC m=+2730.801072816" watchObservedRunningTime="2026-07-24 13:04:17.274817808 +0000 UTC m=+2730.802564865"
Jul 24 13:04:21.375366 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:21.375329    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:04:23.244983 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:23.244951    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:04:28.241178 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:28.241150    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"
Jul 24 13:04:37.205711 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:37.205681    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-c6knq_ca77e87f-a898-4c03-8b7b-ca53cacbf8f7/payload-processing/0.log"
Jul 24 13:04:37.334578 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:37.334546    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7_5a49d324-04b6-4099-b988-a1663eaeb6a5/payload-processing/0.log"
Jul 24 13:04:48.770026 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:04:48.769985    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:05:05.879724 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:05.879689    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:05:29.341574 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:29.341537    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln_19172175-1a85-4f9d-9565-fa066472143c/payload-processing/0.log"
Jul 24 13:05:29.468682 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:29.468643    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-c6knq_ca77e87f-a898-4c03-8b7b-ca53cacbf8f7/payload-processing/0.log"
Jul 24 13:05:29.601889 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:29.601800    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7_5a49d324-04b6-4099-b988-a1663eaeb6a5/payload-processing/0.log"
Jul 24 13:05:29.897781 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:29.897680    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:05:30.218121 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:30.217993    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:05:30.503908 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:30.503858    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:05:32.887061 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:32.887008    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:05:32.981158 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:32.981117    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:05:33.139250 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.139161    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"]
Jul 24 13:05:33.143862 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.143840    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:05:33.146911 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.146883    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-30955e-serving-cert\""
Jul 24 13:05:33.157663 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.157636    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"]
Jul 24 13:05:33.176049 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.175970    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:05:33.250164 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.249662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94fdb95-6d64-40e0-9896-ba6c903f6c61-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt\" (UID: \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:05:33.250164 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.249762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w688v\" (UniqueName: \"kubernetes.io/projected/b94fdb95-6d64-40e0-9896-ba6c903f6c61-kube-api-access-w688v\") pod \"maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt\" (UID: \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:05:33.250164 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.249922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94fdb95-6d64-40e0-9896-ba6c903f6c61-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt\" (UID: \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:05:33.297124 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.297088    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:05:33.350881 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.350843    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94fdb95-6d64-40e0-9896-ba6c903f6c61-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt\" (UID: \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:05:33.351113 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.350911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94fdb95-6d64-40e0-9896-ba6c903f6c61-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt\" (UID: \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:05:33.351113 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.350974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w688v\" (UniqueName: \"kubernetes.io/projected/b94fdb95-6d64-40e0-9896-ba6c903f6c61-kube-api-access-w688v\") pod \"maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt\" (UID: \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:05:33.351658 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.351589    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94fdb95-6d64-40e0-9896-ba6c903f6c61-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt\" (UID: \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:05:33.353646 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.353614    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94fdb95-6d64-40e0-9896-ba6c903f6c61-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt\" (UID: \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:05:33.361751 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.361724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w688v\" (UniqueName: \"kubernetes.io/projected/b94fdb95-6d64-40e0-9896-ba6c903f6c61-kube-api-access-w688v\") pod \"maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt\" (UID: \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:05:33.455832 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.455735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:05:33.605716 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.605688    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"]
Jul 24 13:05:33.609251 ip-10-0-140-61 kubenswrapper[2566]: W0724 13:05:33.609218    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94fdb95_6d64_40e0_9896_ba6c903f6c61.slice/crio-02dd53aa528d8587efbf8093be449ac1feea7f9f68c640f443aa0675e9a798d2 WatchSource:0}: Error finding container 02dd53aa528d8587efbf8093be449ac1feea7f9f68c640f443aa0675e9a798d2: Status 404 returned error can't find the container with id 02dd53aa528d8587efbf8093be449ac1feea7f9f68c640f443aa0675e9a798d2
Jul 24 13:05:33.629895 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.629864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt" event={"ID":"b94fdb95-6d64-40e0-9896-ba6c903f6c61","Type":"ContainerStarted","Data":"02dd53aa528d8587efbf8093be449ac1feea7f9f68c640f443aa0675e9a798d2"}
Jul 24 13:05:33.790891 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.790848    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:05:33.886746 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:33.886684    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:05:34.638236 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:34.638195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt" event={"ID":"b94fdb95-6d64-40e0-9896-ba6c903f6c61","Type":"ContainerStarted","Data":"951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169"}
Jul 24 13:05:34.638770 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:34.638271    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:05:34.658126 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:34.658061    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt" podStartSLOduration=1.212573001 podStartE2EDuration="1.658038931s" totalImagesPullingTime="445.46593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:05:34.656714737 +0000 UTC m=+2808.184461790" watchObservedRunningTime="2026-07-24 13:05:34.658038931 +0000 UTC m=+2808.185785979"
Jul 24 13:05:37.416435 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:37.416402    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:05:40.650563 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:05:40.650532    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:06:09.099839 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.099550    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"]
Jul 24 13:06:09.100499 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.099967    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt" podUID="b94fdb95-6d64-40e0-9896-ba6c903f6c61" containerName="maas-api" containerID="cri-o://951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169" gracePeriod=30
Jul 24 13:06:09.127706 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.127678    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:09.368625 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.368597    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:06:09.488708 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.488678    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w688v\" (UniqueName: \"kubernetes.io/projected/b94fdb95-6d64-40e0-9896-ba6c903f6c61-kube-api-access-w688v\") pod \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\" (UID: \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\") "
Jul 24 13:06:09.488900 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.488739    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94fdb95-6d64-40e0-9896-ba6c903f6c61-maas-api-tls\") pod \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\" (UID: \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\") "
Jul 24 13:06:09.488900 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.488812    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94fdb95-6d64-40e0-9896-ba6c903f6c61-openshift-service-ca\") pod \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\" (UID: \"b94fdb95-6d64-40e0-9896-ba6c903f6c61\") "
Jul 24 13:06:09.489204 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.489157    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94fdb95-6d64-40e0-9896-ba6c903f6c61-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b94fdb95-6d64-40e0-9896-ba6c903f6c61" (UID: "b94fdb95-6d64-40e0-9896-ba6c903f6c61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:06:09.490978 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.490953    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94fdb95-6d64-40e0-9896-ba6c903f6c61-kube-api-access-w688v" (OuterVolumeSpecName: "kube-api-access-w688v") pod "b94fdb95-6d64-40e0-9896-ba6c903f6c61" (UID: "b94fdb95-6d64-40e0-9896-ba6c903f6c61"). InnerVolumeSpecName "kube-api-access-w688v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:06:09.491090 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.491005    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b94fdb95-6d64-40e0-9896-ba6c903f6c61-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b94fdb95-6d64-40e0-9896-ba6c903f6c61" (UID: "b94fdb95-6d64-40e0-9896-ba6c903f6c61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:06:09.590117 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.590082    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94fdb95-6d64-40e0-9896-ba6c903f6c61-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:06:09.590117 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.590108    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94fdb95-6d64-40e0-9896-ba6c903f6c61-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:06:09.590117 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.590119    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w688v\" (UniqueName: \"kubernetes.io/projected/b94fdb95-6d64-40e0-9896-ba6c903f6c61-kube-api-access-w688v\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:06:09.835900 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.835860    2566 generic.go:356] "Generic (PLEG): container finished" podID="b94fdb95-6d64-40e0-9896-ba6c903f6c61" containerID="951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169" exitCode=0
Jul 24 13:06:09.836130 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.835911    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"
Jul 24 13:06:09.836130 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.835951    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt" event={"ID":"b94fdb95-6d64-40e0-9896-ba6c903f6c61","Type":"ContainerDied","Data":"951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169"}
Jul 24 13:06:09.836130 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.836028    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt" event={"ID":"b94fdb95-6d64-40e0-9896-ba6c903f6c61","Type":"ContainerDied","Data":"02dd53aa528d8587efbf8093be449ac1feea7f9f68c640f443aa0675e9a798d2"}
Jul 24 13:06:09.836130 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.836045    2566 scope.go:117] "RemoveContainer" containerID="951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169"
Jul 24 13:06:09.845732 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.845714    2566 scope.go:117] "RemoveContainer" containerID="951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169"
Jul 24 13:06:09.845996 ip-10-0-140-61 kubenswrapper[2566]: E0724 13:06:09.845974    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169\": container with ID starting with 951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169 not found: ID does not exist" containerID="951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169"
Jul 24 13:06:09.846081 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.846005    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169"} err="failed to get container status \"951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169\": rpc error: code = NotFound desc = could not find container \"951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169\": container with ID starting with 951aecbbe509b37175d114cc19aea54f921885c0abe13e00ef1d768db336f169 not found: ID does not exist"
Jul 24 13:06:09.858766 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.858737    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"]
Jul 24 13:06:09.861292 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:09.861270    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-30955e-5bff48fb49-sv4kt"]
Jul 24 13:06:10.976129 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:10.976095    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b94fdb95-6d64-40e0-9896-ba6c903f6c61" path="/var/lib/kubelet/pods/b94fdb95-6d64-40e0-9896-ba6c903f6c61/volumes"
Jul 24 13:06:12.361247 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:12.361198    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:13.721803 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:13.721767    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:14.155697 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:14.155655    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:18.614087 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:18.614051    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:18.713965 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:18.713925    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:19.898309 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:19.898273    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:19.969589 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:19.969549    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:20.163664 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:20.163562    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:20.438161 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:20.438007    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:39.528223 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.527975    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"]
Jul 24 13:06:39.528944 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.528882    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt" podUID="34229da8-f49e-4b19-b1e5-f8ffddb56e02" containerName="maas-api" containerID="cri-o://914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587" gracePeriod=30
Jul 24 13:06:39.554430 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.554389    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:39.581767 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.581728    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"]
Jul 24 13:06:39.581987 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.581953    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln" podUID="19172175-1a85-4f9d-9565-fa066472143c" containerName="payload-processing" containerID="cri-o://a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0" gracePeriod=30
Jul 24 13:06:39.664812 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.664764    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:39.760458 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.760273    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:39.852131 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.852105    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:06:39.885436 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.885410    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"
Jul 24 13:06:39.966812 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.965837    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34229da8-f49e-4b19-b1e5-f8ffddb56e02-maas-api-tls\") pod \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\" (UID: \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\") "
Jul 24 13:06:39.966812 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.965911    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19172175-1a85-4f9d-9565-fa066472143c-plugins-config-volume\") pod \"19172175-1a85-4f9d-9565-fa066472143c\" (UID: \"19172175-1a85-4f9d-9565-fa066472143c\") "
Jul 24 13:06:39.966812 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.965959    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34229da8-f49e-4b19-b1e5-f8ffddb56e02-openshift-service-ca\") pod \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\" (UID: \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\") "
Jul 24 13:06:39.966812 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.966060    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fklq2\" (UniqueName: \"kubernetes.io/projected/19172175-1a85-4f9d-9565-fa066472143c-kube-api-access-fklq2\") pod \"19172175-1a85-4f9d-9565-fa066472143c\" (UID: \"19172175-1a85-4f9d-9565-fa066472143c\") "
Jul 24 13:06:39.966812 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.966143    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6mdb\" (UniqueName: \"kubernetes.io/projected/34229da8-f49e-4b19-b1e5-f8ffddb56e02-kube-api-access-m6mdb\") pod \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\" (UID: \"34229da8-f49e-4b19-b1e5-f8ffddb56e02\") "
Jul 24 13:06:39.967392 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.967278    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34229da8-f49e-4b19-b1e5-f8ffddb56e02-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "34229da8-f49e-4b19-b1e5-f8ffddb56e02" (UID: "34229da8-f49e-4b19-b1e5-f8ffddb56e02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:06:39.967618 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.967578    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19172175-1a85-4f9d-9565-fa066472143c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19172175-1a85-4f9d-9565-fa066472143c" (UID: "19172175-1a85-4f9d-9565-fa066472143c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:06:39.969843 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.969818    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34229da8-f49e-4b19-b1e5-f8ffddb56e02-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "34229da8-f49e-4b19-b1e5-f8ffddb56e02" (UID: "34229da8-f49e-4b19-b1e5-f8ffddb56e02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:06:39.970177 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.970156    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19172175-1a85-4f9d-9565-fa066472143c-kube-api-access-fklq2" (OuterVolumeSpecName: "kube-api-access-fklq2") pod "19172175-1a85-4f9d-9565-fa066472143c" (UID: "19172175-1a85-4f9d-9565-fa066472143c"). InnerVolumeSpecName "kube-api-access-fklq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:06:39.970177 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.970163    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34229da8-f49e-4b19-b1e5-f8ffddb56e02-kube-api-access-m6mdb" (OuterVolumeSpecName: "kube-api-access-m6mdb") pod "34229da8-f49e-4b19-b1e5-f8ffddb56e02" (UID: "34229da8-f49e-4b19-b1e5-f8ffddb56e02"). InnerVolumeSpecName "kube-api-access-m6mdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:06:39.991745 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.991708    2566 generic.go:356] "Generic (PLEG): container finished" podID="19172175-1a85-4f9d-9565-fa066472143c" containerID="a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0" exitCode=0
Jul 24 13:06:39.991914 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.991760    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"
Jul 24 13:06:39.991914 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.991806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln" event={"ID":"19172175-1a85-4f9d-9565-fa066472143c","Type":"ContainerDied","Data":"a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0"}
Jul 24 13:06:39.991914 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.991865    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln" event={"ID":"19172175-1a85-4f9d-9565-fa066472143c","Type":"ContainerDied","Data":"1e1d4610d41109b69e379acf4f7216dafb742b645d64582111ca7aff51228f0e"}
Jul 24 13:06:39.991914 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.991896    2566 scope.go:117] "RemoveContainer" containerID="a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0"
Jul 24 13:06:39.993556 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.993529    2566 generic.go:356] "Generic (PLEG): container finished" podID="34229da8-f49e-4b19-b1e5-f8ffddb56e02" containerID="914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587" exitCode=0
Jul 24 13:06:39.993832 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.993572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt" event={"ID":"34229da8-f49e-4b19-b1e5-f8ffddb56e02","Type":"ContainerDied","Data":"914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587"}
Jul 24 13:06:39.993832 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.993646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt" event={"ID":"34229da8-f49e-4b19-b1e5-f8ffddb56e02","Type":"ContainerDied","Data":"2becc707f041f1dab567a0fb954746f250ee06f355e0c5da43f3dd105350b787"}
Jul 24 13:06:39.993832 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:39.993591    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"
Jul 24 13:06:40.002462 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.002448    2566 scope.go:117] "RemoveContainer" containerID="a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0"
Jul 24 13:06:40.002728 ip-10-0-140-61 kubenswrapper[2566]: E0724 13:06:40.002703    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0\": container with ID starting with a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0 not found: ID does not exist" containerID="a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0"
Jul 24 13:06:40.002846 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.002740    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0"} err="failed to get container status \"a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0\": rpc error: code = NotFound desc = could not find container \"a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0\": container with ID starting with a71ebd0a5fba091d8e2009c454a06e40bb6f4267e11d4d34f9f35de1021f95e0 not found: ID does not exist"
Jul 24 13:06:40.002846 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.002763    2566 scope.go:117] "RemoveContainer" containerID="914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587"
Jul 24 13:06:40.011530 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.011510    2566 scope.go:117] "RemoveContainer" containerID="914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587"
Jul 24 13:06:40.011812 ip-10-0-140-61 kubenswrapper[2566]: E0724 13:06:40.011796    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587\": container with ID starting with 914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587 not found: ID does not exist" containerID="914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587"
Jul 24 13:06:40.011864 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.011818    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587"} err="failed to get container status \"914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587\": rpc error: code = NotFound desc = could not find container \"914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587\": container with ID starting with 914a0c09a1c5ab07526beea652a7d522462b396f1f8339680092a7472b030587 not found: ID does not exist"
Jul 24 13:06:40.017710 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.017687    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"]
Jul 24 13:06:40.023313 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.023278    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-967d3e-849579ff98-j5lln"]
Jul 24 13:06:40.034182 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.034158    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"]
Jul 24 13:06:40.038199 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.038174    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-967d3e-5cdf84bc59-cj5zt"]
Jul 24 13:06:40.067765 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.067683    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19172175-1a85-4f9d-9565-fa066472143c-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:06:40.067765 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.067752    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34229da8-f49e-4b19-b1e5-f8ffddb56e02-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:06:40.068096 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.067775    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fklq2\" (UniqueName: \"kubernetes.io/projected/19172175-1a85-4f9d-9565-fa066472143c-kube-api-access-fklq2\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:06:40.068096 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.067790    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6mdb\" (UniqueName: \"kubernetes.io/projected/34229da8-f49e-4b19-b1e5-f8ffddb56e02-kube-api-access-m6mdb\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:06:40.068096 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.067802    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34229da8-f49e-4b19-b1e5-f8ffddb56e02-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:06:40.976845 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.976758    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19172175-1a85-4f9d-9565-fa066472143c" path="/var/lib/kubelet/pods/19172175-1a85-4f9d-9565-fa066472143c/volumes"
Jul 24 13:06:40.977245 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:40.977220    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34229da8-f49e-4b19-b1e5-f8ffddb56e02" path="/var/lib/kubelet/pods/34229da8-f49e-4b19-b1e5-f8ffddb56e02/volumes"
Jul 24 13:06:44.125320 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:44.125279    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:44.534623 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:44.534590    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:06:49.566031 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:06:49.565990    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:25.657372 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:25.657309    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:26.893153 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:26.893117    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:30.318260 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:30.318224    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:30.457240 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:30.457210    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:32.620754 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:32.620719    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:38.306913 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:38.306867    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:38.373598 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:38.373562    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:38.621371 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:38.621261    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:38.873459 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:38.873360    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:49.682253 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:49.682213    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"]
Jul 24 13:07:49.682801 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:49.682454    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f" podUID="99862a98-6e57-459d-a963-6494863dcf51" containerName="maas-api" containerID="cri-o://25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422" gracePeriod=30
Jul 24 13:07:49.722928 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:49.722893    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:49.742170 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:49.742136    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"]
Jul 24 13:07:49.742392 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:49.742349    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7" podUID="5a49d324-04b6-4099-b988-a1663eaeb6a5" containerName="payload-processing" containerID="cri-o://07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8" gracePeriod=30
Jul 24 13:07:49.845970 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:49.845938    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:49.903001 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:49.902895    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:49.999665 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:49.999639    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:07:50.036181 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.036151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8js6\" (UniqueName: \"kubernetes.io/projected/99862a98-6e57-459d-a963-6494863dcf51-kube-api-access-f8js6\") pod \"99862a98-6e57-459d-a963-6494863dcf51\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") "
Jul 24 13:07:50.036369 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.036209    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99862a98-6e57-459d-a963-6494863dcf51-openshift-service-ca\") pod \"99862a98-6e57-459d-a963-6494863dcf51\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") "
Jul 24 13:07:50.036369 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.036272    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99862a98-6e57-459d-a963-6494863dcf51-maas-api-tls\") pod \"99862a98-6e57-459d-a963-6494863dcf51\" (UID: \"99862a98-6e57-459d-a963-6494863dcf51\") "
Jul 24 13:07:50.036811 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.036780    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99862a98-6e57-459d-a963-6494863dcf51-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99862a98-6e57-459d-a963-6494863dcf51" (UID: "99862a98-6e57-459d-a963-6494863dcf51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:07:50.037369 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.037340    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99862a98-6e57-459d-a963-6494863dcf51-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:07:50.038774 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.038751    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99862a98-6e57-459d-a963-6494863dcf51-kube-api-access-f8js6" (OuterVolumeSpecName: "kube-api-access-f8js6") pod "99862a98-6e57-459d-a963-6494863dcf51" (UID: "99862a98-6e57-459d-a963-6494863dcf51"). InnerVolumeSpecName "kube-api-access-f8js6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:07:50.038855 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.038807    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99862a98-6e57-459d-a963-6494863dcf51-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99862a98-6e57-459d-a963-6494863dcf51" (UID: "99862a98-6e57-459d-a963-6494863dcf51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:07:50.040480 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.040464    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"
Jul 24 13:07:50.138718 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.138678    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqscx\" (UniqueName: \"kubernetes.io/projected/5a49d324-04b6-4099-b988-a1663eaeb6a5-kube-api-access-dqscx\") pod \"5a49d324-04b6-4099-b988-a1663eaeb6a5\" (UID: \"5a49d324-04b6-4099-b988-a1663eaeb6a5\") "
Jul 24 13:07:50.138718 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.138727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a49d324-04b6-4099-b988-a1663eaeb6a5-plugins-config-volume\") pod \"5a49d324-04b6-4099-b988-a1663eaeb6a5\" (UID: \"5a49d324-04b6-4099-b988-a1663eaeb6a5\") "
Jul 24 13:07:50.138954 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.138880    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8js6\" (UniqueName: \"kubernetes.io/projected/99862a98-6e57-459d-a963-6494863dcf51-kube-api-access-f8js6\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:07:50.138954 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.138892    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99862a98-6e57-459d-a963-6494863dcf51-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:07:50.139181 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.139157    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a49d324-04b6-4099-b988-a1663eaeb6a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a49d324-04b6-4099-b988-a1663eaeb6a5" (UID: "5a49d324-04b6-4099-b988-a1663eaeb6a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:07:50.140973 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.140954    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a49d324-04b6-4099-b988-a1663eaeb6a5-kube-api-access-dqscx" (OuterVolumeSpecName: "kube-api-access-dqscx") pod "5a49d324-04b6-4099-b988-a1663eaeb6a5" (UID: "5a49d324-04b6-4099-b988-a1663eaeb6a5"). InnerVolumeSpecName "kube-api-access-dqscx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:07:50.239641 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.239563    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqscx\" (UniqueName: \"kubernetes.io/projected/5a49d324-04b6-4099-b988-a1663eaeb6a5-kube-api-access-dqscx\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:07:50.239641 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.239590    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a49d324-04b6-4099-b988-a1663eaeb6a5-plugins-config-volume\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:07:50.348166 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.348130    2566 generic.go:356] "Generic (PLEG): container finished" podID="99862a98-6e57-459d-a963-6494863dcf51" containerID="25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422" exitCode=0
Jul 24 13:07:50.348331 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.348175    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"
Jul 24 13:07:50.348331 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.348214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f" event={"ID":"99862a98-6e57-459d-a963-6494863dcf51","Type":"ContainerDied","Data":"25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422"}
Jul 24 13:07:50.348331 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.348263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f" event={"ID":"99862a98-6e57-459d-a963-6494863dcf51","Type":"ContainerDied","Data":"3ce4b194c6b8a6c614cdc5a73ac6f2beb4b30f5ce30819c4478370b721e65ea4"}
Jul 24 13:07:50.348331 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.348287    2566 scope.go:117] "RemoveContainer" containerID="25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422"
Jul 24 13:07:50.349685 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.349664    2566 generic.go:356] "Generic (PLEG): container finished" podID="5a49d324-04b6-4099-b988-a1663eaeb6a5" containerID="07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8" exitCode=0
Jul 24 13:07:50.349960 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.349715    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"
Jul 24 13:07:50.349960 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.349746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7" event={"ID":"5a49d324-04b6-4099-b988-a1663eaeb6a5","Type":"ContainerDied","Data":"07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8"}
Jul 24 13:07:50.349960 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.349783    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7" event={"ID":"5a49d324-04b6-4099-b988-a1663eaeb6a5","Type":"ContainerDied","Data":"c8e5747e0efe19ed14cd9e806fe37b68825a647e1d759a20c2b53139c68b1461"}
Jul 24 13:07:50.357211 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.357194    2566 scope.go:117] "RemoveContainer" containerID="25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422"
Jul 24 13:07:50.357478 ip-10-0-140-61 kubenswrapper[2566]: E0724 13:07:50.357459    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422\": container with ID starting with 25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422 not found: ID does not exist" containerID="25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422"
Jul 24 13:07:50.357560 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.357490    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422"} err="failed to get container status \"25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422\": rpc error: code = NotFound desc = could not find container \"25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422\": container with ID starting with 25bd56e03801943397bb5c8c4fe919812d45158c1eb4526681fb19f885d51422 not found: ID does not exist"
Jul 24 13:07:50.357560 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.357514    2566 scope.go:117] "RemoveContainer" containerID="07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8"
Jul 24 13:07:50.366625 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.366608    2566 scope.go:117] "RemoveContainer" containerID="07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8"
Jul 24 13:07:50.366950 ip-10-0-140-61 kubenswrapper[2566]: E0724 13:07:50.366922    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8\": container with ID starting with 07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8 not found: ID does not exist" containerID="07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8"
Jul 24 13:07:50.367001 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.366963    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8"} err="failed to get container status \"07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8\": rpc error: code = NotFound desc = could not find container \"07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8\": container with ID starting with 07bb04372b5ab8908afb5876d5a6715daedf6f88eaa75478bc11d41ab3e808e8 not found: ID does not exist"
Jul 24 13:07:50.375564 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.375540    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"]
Jul 24 13:07:50.381217 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.381192    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-47a7ae-d947bfb6-m2qf7"]
Jul 24 13:07:50.393427 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.393401    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"]
Jul 24 13:07:50.396842 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.396819    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-47a7ae-6b59774c87-t8m8f"]
Jul 24 13:07:50.975836 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.975754    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a49d324-04b6-4099-b988-a1663eaeb6a5" path="/var/lib/kubelet/pods/5a49d324-04b6-4099-b988-a1663eaeb6a5/volumes"
Jul 24 13:07:50.976228 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:50.976169    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99862a98-6e57-459d-a963-6494863dcf51" path="/var/lib/kubelet/pods/99862a98-6e57-459d-a963-6494863dcf51/volumes"
Jul 24 13:07:54.125081 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:54.125012    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:07:54.508772 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:07:54.508735    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:00.382757 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:00.382719    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:00.483182 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:00.483143    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:00.628288 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:00.628254    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:00.884743 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:00.884709    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:11.967729 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:11.967683    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"]
Jul 24 13:08:11.968296 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:11.967953    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9" podUID="ad9a0f38-a4e3-4967-8894-fc5d84c99e54" containerName="maas-api" containerID="cri-o://ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364" gracePeriod=30
Jul 24 13:08:12.000161 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.000125    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:12.100121 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.100072    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:12.200029 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.199978    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:12.238270 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.238246    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 13:08:12.346418 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.346385    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-openshift-service-ca\") pod \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\" (UID: \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\") "
Jul 24 13:08:12.346418 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.346431    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4dt8\" (UniqueName: \"kubernetes.io/projected/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-kube-api-access-n4dt8\") pod \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\" (UID: \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\") "
Jul 24 13:08:12.346683 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.346517    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-maas-api-tls\") pod \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\" (UID: \"ad9a0f38-a4e3-4967-8894-fc5d84c99e54\") "
Jul 24 13:08:12.346788 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.346763    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad9a0f38-a4e3-4967-8894-fc5d84c99e54" (UID: "ad9a0f38-a4e3-4967-8894-fc5d84c99e54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:08:12.348713 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.348686    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-kube-api-access-n4dt8" (OuterVolumeSpecName: "kube-api-access-n4dt8") pod "ad9a0f38-a4e3-4967-8894-fc5d84c99e54" (UID: "ad9a0f38-a4e3-4967-8894-fc5d84c99e54"). InnerVolumeSpecName "kube-api-access-n4dt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:08:12.348829 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.348791    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad9a0f38-a4e3-4967-8894-fc5d84c99e54" (UID: "ad9a0f38-a4e3-4967-8894-fc5d84c99e54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:08:12.448158 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.448118    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:08:12.448158 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.448151    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:08:12.448158 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.448164    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4dt8\" (UniqueName: \"kubernetes.io/projected/ad9a0f38-a4e3-4967-8894-fc5d84c99e54-kube-api-access-n4dt8\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:08:12.463791 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.463748    2566 generic.go:356] "Generic (PLEG): container finished" podID="ad9a0f38-a4e3-4967-8894-fc5d84c99e54" containerID="ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364" exitCode=0
Jul 24 13:08:12.463791 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.463789    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"
Jul 24 13:08:12.464059 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.463830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9" event={"ID":"ad9a0f38-a4e3-4967-8894-fc5d84c99e54","Type":"ContainerDied","Data":"ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364"}
Jul 24 13:08:12.464059 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.463884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9" event={"ID":"ad9a0f38-a4e3-4967-8894-fc5d84c99e54","Type":"ContainerDied","Data":"1fd8020cd87086cdbac7197fc0a10223a7271b2898394b48bde8ac8f37605e19"}
Jul 24 13:08:12.464059 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.463907    2566 scope.go:117] "RemoveContainer" containerID="ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364"
Jul 24 13:08:12.474174 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.474155    2566 scope.go:117] "RemoveContainer" containerID="ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364"
Jul 24 13:08:12.474443 ip-10-0-140-61 kubenswrapper[2566]: E0724 13:08:12.474421    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364\": container with ID starting with ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364 not found: ID does not exist" containerID="ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364"
Jul 24 13:08:12.474492 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.474453    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364"} err="failed to get container status \"ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364\": rpc error: code = NotFound desc = could not find container \"ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364\": container with ID starting with ee481ee1b46c2f67dd0be15907906f39a494a0eed2a8bf2a897c1adca6b75364 not found: ID does not exist"
Jul 24 13:08:12.485644 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.485615    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"]
Jul 24 13:08:12.489504 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.489481    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d15f64-7d8cffb757-grvn9"]
Jul 24 13:08:12.976407 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:12.976374    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad9a0f38-a4e3-4967-8894-fc5d84c99e54" path="/var/lib/kubelet/pods/ad9a0f38-a4e3-4967-8894-fc5d84c99e54/volumes"
Jul 24 13:08:16.138796 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:16.138759    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:16.538709 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:16.538671    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:21.073475 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:21.073441    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:22.707046 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:22.706983    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:22.786731 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:22.786692    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:23.014091 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:23.014056    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:23.303574 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:23.303472    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:33.991960 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:33.991921    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"]
Jul 24 13:08:33.992429 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:33.992231    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv" podUID="7955473c-fa61-4fc1-ac43-ff38e6bf28e9" containerName="maas-api" containerID="cri-o://6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d" gracePeriod=30
Jul 24 13:08:34.019299 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.019260    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:34.089494 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.089464    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:34.189781 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.189746    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:34.256236 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.256214    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 13:08:34.345057 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.345030    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz8ts\" (UniqueName: \"kubernetes.io/projected/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-kube-api-access-rz8ts\") pod \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") "
Jul 24 13:08:34.345258 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.345084    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-openshift-service-ca\") pod \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") "
Jul 24 13:08:34.345258 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.345129    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-maas-api-tls\") pod \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\" (UID: \"7955473c-fa61-4fc1-ac43-ff38e6bf28e9\") "
Jul 24 13:08:34.345453 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.345422    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7955473c-fa61-4fc1-ac43-ff38e6bf28e9" (UID: "7955473c-fa61-4fc1-ac43-ff38e6bf28e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:08:34.347305 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.347279    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-kube-api-access-rz8ts" (OuterVolumeSpecName: "kube-api-access-rz8ts") pod "7955473c-fa61-4fc1-ac43-ff38e6bf28e9" (UID: "7955473c-fa61-4fc1-ac43-ff38e6bf28e9"). InnerVolumeSpecName "kube-api-access-rz8ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:08:34.347402 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.347342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7955473c-fa61-4fc1-ac43-ff38e6bf28e9" (UID: "7955473c-fa61-4fc1-ac43-ff38e6bf28e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:08:34.446626 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.446585    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-openshift-service-ca\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:08:34.446626 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.446621    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-maas-api-tls\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:08:34.446626 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.446630    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz8ts\" (UniqueName: \"kubernetes.io/projected/7955473c-fa61-4fc1-ac43-ff38e6bf28e9-kube-api-access-rz8ts\") on node \"ip-10-0-140-61.ec2.internal\" DevicePath \"\""
Jul 24 13:08:34.577416 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.577311    2566 generic.go:356] "Generic (PLEG): container finished" podID="7955473c-fa61-4fc1-ac43-ff38e6bf28e9" containerID="6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d" exitCode=0
Jul 24 13:08:34.577416 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.577364    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"
Jul 24 13:08:34.577416 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.577396    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv" event={"ID":"7955473c-fa61-4fc1-ac43-ff38e6bf28e9","Type":"ContainerDied","Data":"6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d"}
Jul 24 13:08:34.577695 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.577451    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv" event={"ID":"7955473c-fa61-4fc1-ac43-ff38e6bf28e9","Type":"ContainerDied","Data":"7c11c4c639710d15d8a721478685d1988837fd5778db80cc08e47fd610d9e473"}
Jul 24 13:08:34.577695 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.577473    2566 scope.go:117] "RemoveContainer" containerID="6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d"
Jul 24 13:08:34.589490 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.589467    2566 scope.go:117] "RemoveContainer" containerID="6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d"
Jul 24 13:08:34.589770 ip-10-0-140-61 kubenswrapper[2566]: E0724 13:08:34.589751    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d\": container with ID starting with 6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d not found: ID does not exist" containerID="6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d"
Jul 24 13:08:34.589829 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.589782    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d"} err="failed to get container status \"6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d\": rpc error: code = NotFound desc = could not find container \"6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d\": container with ID starting with 6ff08d5a3f17cc3217d6b47159a690bf4049a9980e8653ecc0a09ffd14dee88d not found: ID does not exist"
Jul 24 13:08:34.602879 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.602850    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"]
Jul 24 13:08:34.606877 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.606847    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6697a-586c6fcdf6-zdgbv"]
Jul 24 13:08:34.975597 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:34.975516    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7955473c-fa61-4fc1-ac43-ff38e6bf28e9" path="/var/lib/kubelet/pods/7955473c-fa61-4fc1-ac43-ff38e6bf28e9/volumes"
Jul 24 13:08:38.498523 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:38.498468    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:38.918672 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:38.918584    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:44.696293 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:44.696256    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:44.796340 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:44.796308    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:44.972683 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:44.972562    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:45.240601 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:45.240488    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:08:47.208869 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:47.208836    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 13:08:47.212894 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:47.212868    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 13:08:51.390330 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:51.390287    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8nn7z_ec482164-b0bd-4a38-993b-489f91fb33d0/manager/0.log"
Jul 24 13:08:51.519980 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:51.519944    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jl9d4_eabaebe5-9634-4a47-a6d8-7fc2d5c980b8/manager/0.log"
Jul 24 13:08:51.657593 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:51.657507    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-cc9bbb694-pvqd5_c4b68762-40f8-4cff-a99b-bb94935bcc07/manager/0.log"
Jul 24 13:08:51.777900 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:51.777868    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qwnnn_fa378298-04b3-4401-8fff-0256174b7a4c/server/0.log"
Jul 24 13:08:51.910699 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:51.910608    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-crq7d_e976a183-bb5d-4441-baa3-2fe8718d5ac1/manager/0.log"
Jul 24 13:08:52.174317 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:52.174226    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c4d787c-skcjs_5571d770-b75d-4adf-aa8e-44ad5860e23f/manager/0.log"
Jul 24 13:08:52.181260 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:52.181230    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c4d787c-skcjs_5571d770-b75d-4adf-aa8e-44ad5860e23f/copy-manifests/0.log"
Jul 24 13:08:52.703077 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:52.703013    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76d9db859c-qnhrd_95c7a827-89e9-4a4b-9227-bd72614db1a8/maas-api/0.log"
Jul 24 13:08:53.176813 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:53.176779    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-jdvqd_d15e8048-1a41-4264-a75a-11389fda4254/postgres/0.log"
Jul 24 13:08:55.950658 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:55.950613    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wrjx5_8674e584-2789-4dfe-b809-9d93b659e554/discovery/0.log"
Jul 24 13:08:56.226464 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:56.226364    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfc95c995-zn967_43656476-8328-40be-9c33-1e1cd7f640f5/kube-auth-proxy/0.log"
Jul 24 13:08:56.473098 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:56.473057    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-mcvc9_3caa1c0e-87cd-4d2b-99f3-06508b01508a/payload-pre-processing/0.log"
Jul 24 13:08:56.599857 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:56.599817    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-c6knq_ca77e87f-a898-4c03-8b7b-ca53cacbf8f7/payload-processing/0.log"
Jul 24 13:08:56.729719 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:56.729683    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86b5bcffdf-j8mkb_7dcef893-f48b-41d9-acd1-50636e761528/router/0.log"
Jul 24 13:08:57.250893 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:57.250858    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-9khb2_848593e9-9746-4e5e-95a7-ce1f8e99d18b/main/0.log"
Jul 24 13:08:57.497679 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:57.497650    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwjwv_7d0f1d38-12ee-4fcf-baf2-4b5fcf462626/main/0.log"
Jul 24 13:08:57.616345 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:08:57.616258    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-g7894_a15675f7-a7d5-43c1-a1ba-ce8eeb0dc6c1/main/0.log"
Jul 24 13:09:01.097416 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:01.097381    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:09:03.322439 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:03.322397    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:09:03.423410 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:03.423367    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log" limit=1024
Jul 24 13:09:06.253048 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:06.252994    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gnrrg_17a4fb33-440c-4563-b7e1-b9ee4ef34d90/global-pull-secret-syncer/0.log"
Jul 24 13:09:06.389287 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:06.389249    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-44h9q_15f50eff-94fd-4dfe-8394-86cca2bfbd41/konnectivity-agent/0.log"
Jul 24 13:09:06.540406 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:06.540306    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-61.ec2.internal_017919978be3c1d1d7d67e195198972d/haproxy/0.log"
Jul 24 13:09:13.595034 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:13.594974    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jlrvs_4dcb8aa8-6eb1-4922-be33-3969dbe5ea51/cluster-monitoring-operator/0.log"
Jul 24 13:09:13.997082 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:13.997050    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xs7tj_c9072e97-484e-4720-b8b7-8339a3ad2d85/node-exporter/0.log"
Jul 24 13:09:14.018810 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:14.018783    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xs7tj_c9072e97-484e-4720-b8b7-8339a3ad2d85/kube-rbac-proxy/0.log"
Jul 24 13:09:14.040301 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:14.040271    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xs7tj_c9072e97-484e-4720-b8b7-8339a3ad2d85/init-textfile/0.log"
Jul 24 13:09:16.012330 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.012299    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-z598v_1d0d7097-31bb-4535-96bc-c30b51af5567/networking-console-plugin/0.log"
Jul 24 13:09:16.626696 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.626662    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/1.log"
Jul 24 13:09:16.635447 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.635419    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfts5_e03b0875-5355-4282-96e5-efe11ccd7618/console-operator/2.log"
Jul 24 13:09:16.723313 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723268    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"]
Jul 24 13:09:16.723669 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723656    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94fdb95-6d64-40e0-9896-ba6c903f6c61" containerName="maas-api"
Jul 24 13:09:16.723718 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723670    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94fdb95-6d64-40e0-9896-ba6c903f6c61" containerName="maas-api"
Jul 24 13:09:16.723718 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723682    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7955473c-fa61-4fc1-ac43-ff38e6bf28e9" containerName="maas-api"
Jul 24 13:09:16.723718 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723688    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7955473c-fa61-4fc1-ac43-ff38e6bf28e9" containerName="maas-api"
Jul 24 13:09:16.723718 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723695    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19172175-1a85-4f9d-9565-fa066472143c" containerName="payload-processing"
Jul 24 13:09:16.723718 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723701    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="19172175-1a85-4f9d-9565-fa066472143c" containerName="payload-processing"
Jul 24 13:09:16.723718 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723710    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99862a98-6e57-459d-a963-6494863dcf51" containerName="maas-api"
Jul 24 13:09:16.723718 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723715    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99862a98-6e57-459d-a963-6494863dcf51" containerName="maas-api"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723722    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34229da8-f49e-4b19-b1e5-f8ffddb56e02" containerName="maas-api"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723727    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="34229da8-f49e-4b19-b1e5-f8ffddb56e02" containerName="maas-api"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723737    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a49d324-04b6-4099-b988-a1663eaeb6a5" containerName="payload-processing"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723742    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a49d324-04b6-4099-b988-a1663eaeb6a5" containerName="payload-processing"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723752    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad9a0f38-a4e3-4967-8894-fc5d84c99e54" containerName="maas-api"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723757    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad9a0f38-a4e3-4967-8894-fc5d84c99e54" containerName="maas-api"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723810    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7955473c-fa61-4fc1-ac43-ff38e6bf28e9" containerName="maas-api"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723817    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="34229da8-f49e-4b19-b1e5-f8ffddb56e02" containerName="maas-api"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723823    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="19172175-1a85-4f9d-9565-fa066472143c" containerName="payload-processing"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723831    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad9a0f38-a4e3-4967-8894-fc5d84c99e54" containerName="maas-api"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723836    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a49d324-04b6-4099-b988-a1663eaeb6a5" containerName="payload-processing"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723841    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94fdb95-6d64-40e0-9896-ba6c903f6c61" containerName="maas-api"
Jul 24 13:09:16.723933 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.723849    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99862a98-6e57-459d-a963-6494863dcf51" containerName="maas-api"
Jul 24 13:09:16.726936 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.726920    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.730027 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.729990    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bz6zh\"/\"default-dockercfg-jsx6c\""
Jul 24 13:09:16.731458 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.731434    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bz6zh\"/\"kube-root-ca.crt\""
Jul 24 13:09:16.731580 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.731457    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bz6zh\"/\"openshift-service-ca.crt\""
Jul 24 13:09:16.731580 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.731537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1d610dd-6adc-4895-b91c-999945611af2-podres\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.731723 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.731618    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74vt2\" (UniqueName: \"kubernetes.io/projected/d1d610dd-6adc-4895-b91c-999945611af2-kube-api-access-74vt2\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.731723 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.731688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1d610dd-6adc-4895-b91c-999945611af2-sys\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.731723 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.731714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1d610dd-6adc-4895-b91c-999945611af2-lib-modules\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.731874 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.731739    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1d610dd-6adc-4895-b91c-999945611af2-proc\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.737189 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.737167    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"]
Jul 24 13:09:16.832356 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.832310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1d610dd-6adc-4895-b91c-999945611af2-podres\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.832559 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.832369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vt2\" (UniqueName: \"kubernetes.io/projected/d1d610dd-6adc-4895-b91c-999945611af2-kube-api-access-74vt2\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.832559 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.832391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1d610dd-6adc-4895-b91c-999945611af2-sys\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.832559 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.832407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1d610dd-6adc-4895-b91c-999945611af2-lib-modules\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.832559 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.832431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1d610dd-6adc-4895-b91c-999945611af2-proc\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.832559 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.832505    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1d610dd-6adc-4895-b91c-999945611af2-proc\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.832559 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.832511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1d610dd-6adc-4895-b91c-999945611af2-sys\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.832559 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.832538    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1d610dd-6adc-4895-b91c-999945611af2-podres\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.832559 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.832559    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1d610dd-6adc-4895-b91c-999945611af2-lib-modules\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:16.841047 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:16.841003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vt2\" (UniqueName: \"kubernetes.io/projected/d1d610dd-6adc-4895-b91c-999945611af2-kube-api-access-74vt2\") pod \"perf-node-gather-daemonset-t5zqv\" (UID: \"d1d610dd-6adc-4895-b91c-999945611af2\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:17.038152 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:17.038122    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:17.175854 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:17.175823    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"]
Jul 24 13:09:17.177770 ip-10-0-140-61 kubenswrapper[2566]: W0724 13:09:17.177742    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd1d610dd_6adc_4895_b91c_999945611af2.slice/crio-9a74ab99c7bd40dd5acd6a487cd247410bbade87c33d7e2bb4769cd30c012993 WatchSource:0}: Error finding container 9a74ab99c7bd40dd5acd6a487cd247410bbade87c33d7e2bb4769cd30c012993: Status 404 returned error can't find the container with id 9a74ab99c7bd40dd5acd6a487cd247410bbade87c33d7e2bb4769cd30c012993
Jul 24 13:09:17.796539 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:17.796499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv" event={"ID":"d1d610dd-6adc-4895-b91c-999945611af2","Type":"ContainerStarted","Data":"e9965de5920aebed0fb572392794fe6879468d6efc3791c3a0e3e8cc362cbce0"}
Jul 24 13:09:17.796743 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:17.796556    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv" event={"ID":"d1d610dd-6adc-4895-b91c-999945611af2","Type":"ContainerStarted","Data":"9a74ab99c7bd40dd5acd6a487cd247410bbade87c33d7e2bb4769cd30c012993"}
Jul 24 13:09:17.796743 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:17.796594    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:17.815749 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:17.815697    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv" podStartSLOduration=1.8156802760000001 podStartE2EDuration="1.815680276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:09:17.814398946 +0000 UTC m=+3031.342146003" watchObservedRunningTime="2026-07-24 13:09:17.815680276 +0000 UTC m=+3031.343427377"
Jul 24 13:09:17.942941 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:17.942901    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-k6xvq_0c0e4621-d282-4a07-b820-9df31d6ce258/volume-data-source-validator/0.log"
Jul 24 13:09:19.097820 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:19.097784    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rxswx_94cc8f44-c20d-472c-a11d-986174072f55/dns/0.log"
Jul 24 13:09:19.120107 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:19.120077    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rxswx_94cc8f44-c20d-472c-a11d-986174072f55/kube-rbac-proxy/0.log"
Jul 24 13:09:19.168588 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:19.168562    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hjxk2_a57be8b6-3710-4d2c-a3f8-e5c7b5f89ba3/dns-node-resolver/0.log"
Jul 24 13:09:19.778211 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:19.778178    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n6d7j_f905cc3c-2a1f-4cfc-934d-e465550ec974/node-ca/0.log"
Jul 24 13:09:21.021494 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:21.021464    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wrjx5_8674e584-2789-4dfe-b809-9d93b659e554/discovery/0.log"
Jul 24 13:09:21.066003 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:21.065961    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfc95c995-zn967_43656476-8328-40be-9c33-1e1cd7f640f5/kube-auth-proxy/0.log"
Jul 24 13:09:21.163584 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:21.163554    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-mcvc9_3caa1c0e-87cd-4d2b-99f3-06508b01508a/payload-pre-processing/0.log"
Jul 24 13:09:21.202069 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:21.202034    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-c6knq_ca77e87f-a898-4c03-8b7b-ca53cacbf8f7/payload-processing/0.log"
Jul 24 13:09:21.228794 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:21.228765    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86b5bcffdf-j8mkb_7dcef893-f48b-41d9-acd1-50636e761528/router/0.log"
Jul 24 13:09:22.073756 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:22.073727    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nfjm7_d6224e74-72f6-4d5c-bbbd-d87bb40d9507/serve-healthcheck-canary/0.log"
Jul 24 13:09:22.573794 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:22.573756    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-zlm49_45ae8a84-d47e-461b-bd9a-6d076a5b94f0/insights-operator/0.log"
Jul 24 13:09:22.575050 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:22.574996    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-zlm49_45ae8a84-d47e-461b-bd9a-6d076a5b94f0/insights-operator/1.log"
Jul 24 13:09:22.596733 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:22.596697    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9swnm_76b6e7bc-a18d-4155-aaa9-11a3a084be66/kube-rbac-proxy/0.log"
Jul 24 13:09:22.619530 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:22.619498    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9swnm_76b6e7bc-a18d-4155-aaa9-11a3a084be66/exporter/0.log"
Jul 24 13:09:22.642115 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:22.642090    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9swnm_76b6e7bc-a18d-4155-aaa9-11a3a084be66/extractor/0.log"
Jul 24 13:09:23.812438 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:23.812405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-t5zqv"
Jul 24 13:09:24.867884 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:24.867846    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8nn7z_ec482164-b0bd-4a38-993b-489f91fb33d0/manager/0.log"
Jul 24 13:09:25.235116 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:25.235011    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jl9d4_eabaebe5-9634-4a47-a6d8-7fc2d5c980b8/manager/0.log"
Jul 24 13:09:25.467899 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:25.467854    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-cc9bbb694-pvqd5_c4b68762-40f8-4cff-a99b-bb94935bcc07/manager/0.log"
Jul 24 13:09:25.489321 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:25.489240    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qwnnn_fa378298-04b3-4401-8fff-0256174b7a4c/server/0.log"
Jul 24 13:09:25.608381 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:25.608348    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-crq7d_e976a183-bb5d-4441-baa3-2fe8718d5ac1/manager/0.log"
Jul 24 13:09:25.797382 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:25.797349    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c4d787c-skcjs_5571d770-b75d-4adf-aa8e-44ad5860e23f/manager/0.log"
Jul 24 13:09:25.818011 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:25.817977    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c4d787c-skcjs_5571d770-b75d-4adf-aa8e-44ad5860e23f/copy-manifests/0.log"
Jul 24 13:09:27.645431 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:27.645395    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wrjx5_8674e584-2789-4dfe-b809-9d93b659e554/discovery/0.log"
Jul 24 13:09:27.697009 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:27.696977    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfc95c995-zn967_43656476-8328-40be-9c33-1e1cd7f640f5/kube-auth-proxy/0.log"
Jul 24 13:09:27.796694 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:27.796656    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-mcvc9_3caa1c0e-87cd-4d2b-99f3-06508b01508a/payload-pre-processing/0.log"
Jul 24 13:09:27.832490 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:27.832456    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-c6knq_ca77e87f-a898-4c03-8b7b-ca53cacbf8f7/payload-processing/0.log"
Jul 24 13:09:27.847327 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:27.847296    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86b5bcffdf-j8mkb_7dcef893-f48b-41d9-acd1-50636e761528/router/0.log"
Jul 24 13:09:29.517784 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:29.517750    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wrjx5_8674e584-2789-4dfe-b809-9d93b659e554/discovery/0.log"
Jul 24 13:09:29.562108 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:29.562047    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfc95c995-zn967_43656476-8328-40be-9c33-1e1cd7f640f5/kube-auth-proxy/0.log"
Jul 24 13:09:29.651263 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:29.651226    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-mcvc9_3caa1c0e-87cd-4d2b-99f3-06508b01508a/payload-pre-processing/0.log"
Jul 24 13:09:29.686705 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:29.686667    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-c6knq_ca77e87f-a898-4c03-8b7b-ca53cacbf8f7/payload-processing/0.log"
Jul 24 13:09:29.707883 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:29.707853    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86b5bcffdf-j8mkb_7dcef893-f48b-41d9-acd1-50636e761528/router/0.log"
Jul 24 13:09:32.815623 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:32.815579    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n_f810f78d-7227-4bd0-aab4-194ced389280/kube-storage-version-migrator-operator/1.log"
Jul 24 13:09:32.817381 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:32.817353    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-pz57n_f810f78d-7227-4bd0-aab4-194ced389280/kube-storage-version-migrator-operator/0.log"
Jul 24 13:09:34.100667 ip-10-0-140-61 kubenswrapper[2566]: I0724 13:09:34.100635    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mm5x2_32e6a712-727d-4045-927a-dd28b210caba/kube-multus-additional-cni-plugins/0.log"