Jul 24 17:36:26.387892 ip-10-0-141-44 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 17:36:26.715419 ip-10-0-141-44 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 17:36:26.715419 ip-10-0-141-44 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 17:36:26.715419 ip-10-0-141-44 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 17:36:26.715419 ip-10-0-141-44 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 17:36:26.715419 ip-10-0-141-44 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 17:36:26.716961 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.716884    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 17:36:26.720841 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720826    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:36:26.720841 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720839    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720845    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720850    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720853    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720856    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720858    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720861    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720864    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720867    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720870    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720872    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720874    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720878    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720886    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720889    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720891    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720894    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720896    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720899    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720902    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:36:26.720904 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720905    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720907    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720910    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720913    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720917    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720919    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720922    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720924    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720927    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720929    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720932    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720934    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720936    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720939    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720943    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720945    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720948    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720950    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720953    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720955    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:36:26.721372 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720958    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720960    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720963    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720965    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720967    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720970    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720972    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720975    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720977    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720979    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720982    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720985    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720987    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720990    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720993    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720996    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.720998    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721001    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721004    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721007    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:36:26.721883 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721010    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721012    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721015    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721017    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721020    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721022    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721025    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721027    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721031    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721035    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721038    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721041    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721044    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721046    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721050    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721052    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721055    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721059    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721063    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:36:26.722368 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721066    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721069    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721071    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721075    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721078    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721081    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721083    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721086    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721088    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721091    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721093    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721096    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721099    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721101    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721104    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721106    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721109    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721111    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:36:26.722851 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.721114    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:36:26.723998 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.723985    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:36:26.723998 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.723997    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724001    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724004    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724006    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724009    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724013    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724016    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724019    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724022    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724024    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724027    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724029    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724032    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724034    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724037    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724039    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724042    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724044    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724047    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724050    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:36:26.724061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724052    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724055    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724057    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724060    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724063    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724065    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724068    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724072    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724074    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724077    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724080    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724082    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724085    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724087    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724090    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724092    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724095    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724097    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724100    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:36:26.724604 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724103    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724105    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724108    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724110    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724113    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724115    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724118    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724120    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724124    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724127    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724130    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724133    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724135    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724138    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724142    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724146    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724149    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724152    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724155    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:36:26.725061 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724158    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724161    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724164    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724167    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724170    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724173    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724175    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724178    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724180    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724182    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724185    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724187    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724190    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724193    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724195    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724198    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724200    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724203    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724206    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724208    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:36:26.725528 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724211    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724213    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724216    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724218    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724221    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724223    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724226    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724229    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724232    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724235    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724237    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724240    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724244    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724247    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724250    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724253    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724256    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724259    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724261    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724264    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:36:26.726024 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724625    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724630    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724632    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724635    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724639    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724642    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724645    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724647    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724650    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724653    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724655    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724658    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724660    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724663    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724666    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724668    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724671    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724673    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724676    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724679    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:36:26.726629 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724682    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724685    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724688    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724690    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724693    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724696    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724698    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724701    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724703    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724706    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724709    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724711    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724714    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724716    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724719    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724721    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724724    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724727    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724730    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724732    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:36:26.727110 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724735    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724737    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724739    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724742    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724745    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724747    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724750    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724752    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724755    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724758    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724764    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724766    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724769    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724772    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724774    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724777    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724779    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724782    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724784    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:36:26.727609 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724787    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724790    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724793    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724796    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724798    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724800    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724803    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724805    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724809    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724813    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724816    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724819    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724822    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724825    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724827    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724830    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724833    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724836    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724839    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:36:26.728063 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724842    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724845    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724848    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724851    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724855    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724857    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724860    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724862    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724865    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724867    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724870    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724872    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724875    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724879    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724882    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724886    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724888    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724891    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724894    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:36:26.728518 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724896    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.724899    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.724963    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.724969    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.724975    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.724979    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.724997    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725002    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725008    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725015    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725019    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725022    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725026    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725029    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725032    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725036    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725039    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725044    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725047    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725050    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725054    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725056    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725060    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725063    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 17:36:26.729018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725066    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725071    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725074    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725077    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725081    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725084    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725086    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725090    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725093    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725096    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725100    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725103    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725105    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725108    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725111    2581 flags.go:64] FLAG: --enable-server="true"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725115    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725551    2581 flags.go:64] FLAG: --event-burst="100"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725556    2581 flags.go:64] FLAG: --event-qps="50"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725559    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725562    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725579    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725583    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725586    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725589    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725592    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 24 17:36:26.729635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725596    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725599    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725602    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725605    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725608    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725611    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725614    2581 flags.go:64] FLAG: --feature-gates=""
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725618    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725621    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725624    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725627    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725630    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725633    2581 flags.go:64] FLAG: --help="false"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725636    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-141-44.ec2.internal"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725639    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725642    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725645    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725649    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725652    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725655    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725658    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725661    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725665    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 17:36:26.730234 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725669    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725673    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725677    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725679    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725682    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725685    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725688    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725692    2581 flags.go:64] FLAG: --lock-file=""
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725695    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725698    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725701    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725706    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725709    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725712    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725715    2581 flags.go:64] FLAG: --logging-format="text"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725718    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725721    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725724    2581 flags.go:64] FLAG: --manifest-url=""
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725727    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725731    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725734    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725738    2581 flags.go:64] FLAG: --max-pods="110"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725741    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725744    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725747    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 17:36:26.730824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725750    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725753    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725756    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725758    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725765    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725768    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725772    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725775    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725778    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725784    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725787    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725790    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725793    2581 flags.go:64] FLAG: --port="10250"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725796    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725798    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f9b957474d84083b"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725801    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725805    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725807    2581 flags.go:64] FLAG: --register-node="true"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725810    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725814    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725817    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725820    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725823    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725826    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725829    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 17:36:26.731458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725832    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725835    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725838    2581 flags.go:64] FLAG: --runonce="false"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725841    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725844    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725847    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725850    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725852    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725856    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725859    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725862    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725864    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725867    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725870    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725874    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725877    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725881    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725884    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725889    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725893    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725896    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725900    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725903    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725906    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725909    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 17:36:26.732088 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725912    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725915    2581 flags.go:64] FLAG: --v="2"
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725919    2581 flags.go:64] FLAG: --version="false"
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725923    2581 flags.go:64] FLAG: --vmodule=""
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725926    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.725929    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726045    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726050    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726053    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726056    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726059    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726063    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726067    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726070    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726072    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726075    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726078    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726081    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726084    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726086    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:36:26.732738 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726089    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726095    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726099    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726103    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726107    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726110    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726113    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726116    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726119    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726121    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726124    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726127    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726129    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726132    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726134    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726137    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726140    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726142    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726145    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726147    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:36:26.733221 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726150    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726152    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726155    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726157    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726160    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726162    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726165    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726167    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726170    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726173    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726175    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726177    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726180    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726184    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726186    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726189    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726191    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726194    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726198    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726201    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:36:26.733724 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726203    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726206    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726208    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726212    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726215    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726219    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726221    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726224    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726227    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726230    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726232    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726235    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726238    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726240    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726243    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726245    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726248    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726250    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726253    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:36:26.734200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726255    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726258    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726260    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726263    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726265    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726268    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726271    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726274    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726277    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726279    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726282    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726286    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726288    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726291    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726294    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726296    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726299    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726301    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726304    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726306    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:36:26.734722 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726309    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726311    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726314    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726317    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726319    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.726322    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.726330    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.733241    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.733258    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.733962    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.733972    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734046    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734054    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734058    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:36:26.735200 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734063    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734067    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734070    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734074    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734078    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734082    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734086    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734090    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734093    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734099    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734103    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734106    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734110    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734114    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734118    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734121    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734125    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734128    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734132    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734137    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:36:26.735551 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734141    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734145    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734148    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734152    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734155    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734160    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734164    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734167    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734172    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734175    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734179    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734183    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734187    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734191    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734194    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734198    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734201    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734205    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734210    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734215    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:36:26.736047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734218    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734222    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734226    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734232    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734239    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734244    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734250    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734255    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734260    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734264    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734268    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734272    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734276    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734282    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734287    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734291    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734296    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734300    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734309    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:36:26.736538 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734313    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734318    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734322    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734326    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734329    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734333    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734337    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734340    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734344    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734348    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734352    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734355    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734359    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734362    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734366    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734370    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734374    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734378    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734381    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734385    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:36:26.737005 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734388    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734392    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734396    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734399    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734402    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734406    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734409    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734413    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734417    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734421    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734424    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734428    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734432    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734435    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734439    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734443    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:36:26.737602 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734447    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.734453    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734616    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734623    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734627    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734630    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734634    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734638    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734642    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734646    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734649    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734653    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734657    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734662    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:36:26.738331 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734666    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734670    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734674    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734677    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734681    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734685    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734689    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734692    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734695    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734700    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734704    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734708    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734712    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734716    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734719    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734723    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734727    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734731    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734734    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:36:26.738964 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734739    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734742    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734746    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734750    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734755    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734760    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734764    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734768    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734772    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734776    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734780    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734784    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734788    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734792    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734796    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734799    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734803    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734807    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734810    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:36:26.739592 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734814    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734817    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734821    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734825    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734828    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734833    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734836    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734840    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734843    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734847    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734850    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734854    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734858    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734861    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734865    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734869    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734873    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734876    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734881    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734884    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:36:26.740100 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734888    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734891    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734894    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734898    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734902    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734906    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734909    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734915    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734919    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734924    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734928    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734932    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734936    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734940    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734944    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734948    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734952    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734956    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734960    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734963    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:36:26.740620 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734967    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734971    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734975    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734978    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734982    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734985    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734989    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734992    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:26.734996    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.735003    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.735127    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.739297    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.740690    2581 server.go:1019] "Starting client certificate rotation"
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.740794    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 17:36:26.741184 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.740830    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 17:36:26.742995 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.742976    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 17:36:26.744106 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.744089    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 17:36:26.752073 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.752053    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 24 17:36:26.754157 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.754141    2581 log.go:25] "Validated CRI v1 image API"
Jul 24 17:36:26.754926 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.754912    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 17:36:26.757388 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.757371    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 940be7ad-ef67-4e9e-b892-223242509b37:/dev/nvme0n1p4 a839d9af-1d57-465c-b652-6463ceb1e132:/dev/nvme0n1p3]
Jul 24 17:36:26.757436 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.757388    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 17:36:26.762428 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.762319    2581 manager.go:216] Machine: {Timestamp:2026-07-24 17:36:26.76110679 +0000 UTC m=+0.277617818 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107745 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec279ea4e6657e56916b7fb947abe4c6 SystemUUID:ec279ea4-e665-7e56-916b-7fb947abe4c6 BootID:c05af4a3-9266-4ef5-9ed3-df02df41e675 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c4:ec:f5:56:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c4:ec:f5:56:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:2c:96:e8:70:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 17:36:26.762428 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.762427    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 17:36:26.762537 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.762498    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 17:36:26.764651 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.764617    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 17:36:26.764854 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.764659    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-44.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 17:36:26.765378 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.765364    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 17:36:26.765437 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.765385    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 17:36:26.765437 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.765398    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 17:36:26.765437 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.765419    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 17:36:26.766377 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.766364    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 17:36:26.769548 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.769533    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 17:36:26.769689 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.769678    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 17:36:26.770341 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.770330    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 17:36:26.770399 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.770347    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 17:36:26.770399 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.770366    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 17:36:26.770399 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.770380    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 24 17:36:26.770399 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.770393    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 17:36:26.770798 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.770782    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 17:36:26.770865 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.770809    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 17:36:26.772350 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.772333    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 17:36:26.772724 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.772705    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 17:36:26.776109 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.776091    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 17:36:26.776174 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.776133    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 17:36:26.777739 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.777725    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 17:36:26.777795 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.777747    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 17:36:26.777795 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.777756    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 17:36:26.777795 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.777765    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 17:36:26.777795 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.777777    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 17:36:26.777795 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.777785    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 17:36:26.777934 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.777797    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 17:36:26.777934 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.777806    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 17:36:26.777934 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.777816    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 17:36:26.777934 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.777824    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 17:36:26.777934 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.777836    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 17:36:26.777934 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.777843    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 17:36:26.779138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.779128    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 17:36:26.779180 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.779142    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 17:36:26.782691 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.782404    2581 server.go:1268] "Started kubelet"
Jul 24 17:36:26.782691 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.782481    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-44.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 17:36:26.782691 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.782489    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-44.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 17:36:26.782691 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.782487    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 17:36:26.782691 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.782548    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 17:36:26.782691 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.782522    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 17:36:26.782691 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.782608    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 17:36:26.783249 ip-10-0-141-44 systemd[1]: Started Kubernetes Kubelet.
Jul 24 17:36:26.784244 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.784206    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 17:36:26.784296 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.784282    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 17:36:26.788156 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.788128    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 17:36:26.790654 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.790636    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 17:36:26.790753 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.790640    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 17:36:26.790753 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.790736    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 24 17:36:26.791325 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.791137    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 17:36:26.791325 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.791189    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 17:36:26.791325 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.791205    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 17:36:26.791325 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.791289    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 17:36:26.791325 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.791308    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 17:36:26.791810 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.791778    2581 factory.go:55] Registering systemd factory
Jul 24 17:36:26.791810 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.791796    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 24 17:36:26.792219 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.791444    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-44.ec2.internal.18c54a5c1b93c4af  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-44.ec2.internal,UID:ip-10-0-141-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-44.ec2.internal,},FirstTimestamp:2026-07-24 17:36:26.782385327 +0000 UTC m=+0.298896354,LastTimestamp:2026-07-24 17:36:26.782385327 +0000 UTC m=+0.298896354,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-44.ec2.internal,}"
Jul 24 17:36:26.793016 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.792998    2581 factory.go:153] Registering CRI-O factory
Jul 24 17:36:26.793016 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.793015    2581 factory.go:223] Registration of the crio container factory successfully
Jul 24 17:36:26.793148 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.793060    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 17:36:26.793148 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.793081    2581 factory.go:103] Registering Raw factory
Jul 24 17:36:26.793148 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.793094    2581 manager.go:1196] Started watching for new ooms in manager
Jul 24 17:36:26.793474 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.793462    2581 manager.go:349] Starting recovery of all containers
Jul 24 17:36:26.801764 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.801738    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 17:36:26.801838 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.801761    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 17:36:26.801838 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.801812    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-44.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 17:36:26.807763 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.807747    2581 manager.go:354] Recovery completed
Jul 24 17:36:26.811741 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.811725    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:36:26.813498 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.813475    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:36:26.813552 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.813512    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:36:26.813552 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.813526    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:36:26.814008 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.813995    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 17:36:26.814008 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.814007    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 17:36:26.814129 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.814011    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:36:26.814129 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.814025    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 17:36:26.814715 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.814698    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:36:26.814774 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.814726    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:36:26.814774 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.814736    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:36:26.815999 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.815936    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-44.ec2.internal.18c54a5c1d6e8375  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-44.ec2.internal,UID:ip-10-0-141-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-44.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-44.ec2.internal,},FirstTimestamp:2026-07-24 17:36:26.813498229 +0000 UTC m=+0.330009259,LastTimestamp:2026-07-24 17:36:26.813498229 +0000 UTC m=+0.330009259,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-44.ec2.internal,}"
Jul 24 17:36:26.816674 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.816661    2581 policy_none.go:49] "None policy: Start"
Jul 24 17:36:26.816733 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.816679    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 17:36:26.816733 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.816694    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 17:36:26.818713 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.818700    2581 policy_none.go:47] "Start"
Jul 24 17:36:26.826228 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.826079    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-44.ec2.internal.18c54a5c1d6ed5f6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-44.ec2.internal,UID:ip-10-0-141-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-44.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-44.ec2.internal,},FirstTimestamp:2026-07-24 17:36:26.81351935 +0000 UTC m=+0.330030379,LastTimestamp:2026-07-24 17:36:26.81351935 +0000 UTC m=+0.330030379,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-44.ec2.internal,}"
Jul 24 17:36:26.840082 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.840016    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-44.ec2.internal.18c54a5c1d6efe6e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-44.ec2.internal,UID:ip-10-0-141-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-44.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-44.ec2.internal,},FirstTimestamp:2026-07-24 17:36:26.81352971 +0000 UTC m=+0.330040738,LastTimestamp:2026-07-24 17:36:26.81352971 +0000 UTC m=+0.330040738,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-44.ec2.internal,}"
Jul 24 17:36:26.840628 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.840610    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mtn7w"
Jul 24 17:36:26.851273 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.851255    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mtn7w"
Jul 24 17:36:26.852164 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.852101    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-44.ec2.internal.18c54a5c1d6e8375\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-44.ec2.internal.18c54a5c1d6e8375  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-44.ec2.internal,UID:ip-10-0-141-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-44.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-44.ec2.internal,},FirstTimestamp:2026-07-24 17:36:26.813498229 +0000 UTC m=+0.330009259,LastTimestamp:2026-07-24 17:36:26.814714338 +0000 UTC m=+0.331225364,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-44.ec2.internal,}"
Jul 24 17:36:26.858123 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.858109    2581 manager.go:337] "Starting Device Plugin manager"
Jul 24 17:36:26.858189 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.858135    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 17:36:26.858189 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.858153    2581 server.go:87] "Starting device plugin registration server"
Jul 24 17:36:26.858350 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.858338    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 17:36:26.858421 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.858352    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 17:36:26.858479 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.858452    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 17:36:26.858540 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.858532    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 17:36:26.858608 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.858541    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 17:36:26.859042 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.859024    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 17:36:26.859123 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.859068    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 24 17:36:26.951609 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.951585    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 17:36:26.953684 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.951625    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 17:36:26.953684 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.951646    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 17:36:26.953684 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.951684    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 17:36:26.954430 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.954412    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:36:26.958820 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.958808    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:36:26.960311 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.960297    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:36:26.960375 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.960321    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:36:26.960375 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.960331    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:36:26.960375 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.960352    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-44.ec2.internal"
Jul 24 17:36:26.968192 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:26.968153    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-44.ec2.internal"
Jul 24 17:36:26.968192 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:26.968173    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-44.ec2.internal\": node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 24 17:36:27.006973 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:27.006953    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 24 17:36:27.052586 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.052544    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"]
Jul 24 17:36:27.052670 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.052629    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:36:27.053486 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.053473    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:36:27.053565 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.053501    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:36:27.053565 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.053512    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:36:27.055802 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.055791    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:36:27.055912 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.055898    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.055959 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.055925    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:36:27.056519 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.056501    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:36:27.056619 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.056509    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:36:27.056619 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.056553    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:36:27.056619 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.056564    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:36:27.056619 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.056529    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:36:27.056619 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.056605    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:36:27.058764 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.058751    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.058813 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.058776    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:36:27.059521 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.059505    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:36:27.059611 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.059534    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:36:27.059611 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.059548    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:36:27.082254 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:27.082235    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-44.ec2.internal\" not found" node="ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.087071 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:27.087053    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-44.ec2.internal\" not found" node="ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.093849 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.093832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/780251891b48cdf9ed19946decb1fe94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal\" (UID: \"780251891b48cdf9ed19946decb1fe94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.093899 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.093857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/780251891b48cdf9ed19946decb1fe94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal\" (UID: \"780251891b48cdf9ed19946decb1fe94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.093899 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.093873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc5bec91994039b1fb9327533b8ad0c7-config\") pod \"kube-apiserver-proxy-ip-10-0-141-44.ec2.internal\" (UID: \"fc5bec91994039b1fb9327533b8ad0c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.107612 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:27.107595    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 24 17:36:27.194333 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.194313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/780251891b48cdf9ed19946decb1fe94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal\" (UID: \"780251891b48cdf9ed19946decb1fe94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.194405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.194363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/780251891b48cdf9ed19946decb1fe94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal\" (UID: \"780251891b48cdf9ed19946decb1fe94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.194405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.194336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/780251891b48cdf9ed19946decb1fe94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal\" (UID: \"780251891b48cdf9ed19946decb1fe94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.194405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.194393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/780251891b48cdf9ed19946decb1fe94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal\" (UID: \"780251891b48cdf9ed19946decb1fe94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.194501 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.194429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc5bec91994039b1fb9327533b8ad0c7-config\") pod \"kube-apiserver-proxy-ip-10-0-141-44.ec2.internal\" (UID: \"fc5bec91994039b1fb9327533b8ad0c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.194501 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.194438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc5bec91994039b1fb9327533b8ad0c7-config\") pod \"kube-apiserver-proxy-ip-10-0-141-44.ec2.internal\" (UID: \"fc5bec91994039b1fb9327533b8ad0c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.208429 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:27.208409    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 24 17:36:27.309054 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:27.309032    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 24 17:36:27.386199 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.386170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.390736 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.390721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.409521 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:27.409503    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 24 17:36:27.509870 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:27.509846    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 24 17:36:27.610335 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:27.610281    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 24 17:36:27.710695 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:27.710661    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 24 17:36:27.739727 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.739700    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 17:36:27.740077 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.739834    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 17:36:27.811085 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:27.811064    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 24 17:36:27.856254 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.856231    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 17:31:26 +0000 UTC" deadline="2028-04-25 17:02:03.573584098 +0000 UTC"
Jul 24 17:36:27.856254 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.856252    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15383h25m35.717335429s"
Jul 24 17:36:27.857648 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.857634    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:36:27.883049 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.883007    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:36:27.890810 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.890789    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.904939 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.904925    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 17:36:27.906273 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.906260    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"
Jul 24 17:36:27.916062 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:27.916048    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 17:36:28.032621 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:28.032591    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780251891b48cdf9ed19946decb1fe94.slice/crio-0554d492a0c8c89c2d1b8e6b5157d3e03f0421ff5102636d59b924197d74467e WatchSource:0}: Error finding container 0554d492a0c8c89c2d1b8e6b5157d3e03f0421ff5102636d59b924197d74467e: Status 404 returned error can't find the container with id 0554d492a0c8c89c2d1b8e6b5157d3e03f0421ff5102636d59b924197d74467e
Jul 24 17:36:28.033047 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:28.033022    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5bec91994039b1fb9327533b8ad0c7.slice/crio-10984b5a7814a2d7be693fd3c8d8b9c7524ad981a47163659ce7e4ad3f56feec WatchSource:0}: Error finding container 10984b5a7814a2d7be693fd3c8d8b9c7524ad981a47163659ce7e4ad3f56feec: Status 404 returned error can't find the container with id 10984b5a7814a2d7be693fd3c8d8b9c7524ad981a47163659ce7e4ad3f56feec
Jul 24 17:36:28.037724 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.037706    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:36:28.125366 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.125339    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:36:28.747949 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.747920    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:36:28.771941 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.771915    2581 apiserver.go:52] "Watching apiserver"
Jul 24 17:36:28.778733 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.778703    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 17:36:28.779085 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.779064    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-9l4cq","openshift-ovn-kubernetes/ovnkube-node-t6krb","kube-system/global-pull-secret-syncer-zdzrg","kube-system/konnectivity-agent-8p2dj","kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rddq7","openshift-image-registry/node-ca-gnvrc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal","openshift-multus/multus-additional-cni-plugins-6h2l8","openshift-network-diagnostics/network-check-target-58pqw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8","openshift-dns/node-resolver-r9gdx","openshift-multus/multus-7pvf9","openshift-multus/network-metrics-daemon-6fck2"]
Jul 24 17:36:28.781260 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.781240    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:28.781366 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:28.781314    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:28.783160 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.783136    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.785582 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.785543    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 17:36:28.785735 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.785718    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dvz5p\""
Jul 24 17:36:28.785829 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.785810    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 17:36:28.786000 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.785945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.788023 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.788004    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8p2dj"
Jul 24 17:36:28.788412 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.788216    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 17:36:28.788412 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.788319    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-trs78\""
Jul 24 17:36:28.788412 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.788338    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 17:36:28.788611 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.788591    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 17:36:28.790273 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.789866    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pkxb6\""
Jul 24 17:36:28.790273 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.790152    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.790855 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.790833    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 17:36:28.790855 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.790850    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 17:36:28.790977 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.790844    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 17:36:28.792048 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.792032    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 17:36:28.792129 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.792115    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 17:36:28.792441 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.792430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.792517 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.792505    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 17:36:28.793810 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.793792    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 17:36:28.794028 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.794008    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rzdqp\""
Jul 24 17:36:28.794119 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.794008    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 17:36:28.794789 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.794772    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.794949 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.794836    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 17:36:28.796490 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.796470    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 17:36:28.797115 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.796758    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 17:36:28.797115 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.797073    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gnvrc"
Jul 24 17:36:28.797478 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.797462    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 17:36:28.797597 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.797516    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 17:36:28.797685 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.797558    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 17:36:28.797754 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.797684    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nm6jb\""
Jul 24 17:36:28.797754 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.797728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x7qwt\""
Jul 24 17:36:28.798222 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.798205    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 17:36:28.799099 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.799083    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9l4cq"
Jul 24 17:36:28.800031 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.800016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 17:36:28.800367 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.800350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v6t7j\""
Jul 24 17:36:28.800434 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.800354    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 17:36:28.800706 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.800692    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 17:36:28.801322 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.801305    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:28.801402 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:28.801381    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:28.802470 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.802453    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 17:36:28.803345 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.803329    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 17:36:28.804723 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.804695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-node-log\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.804849 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.804741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.804906 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.804869    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7df3a483-f195-40a9-b1d5-b13f47f41dc3-cnibin\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.804906 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.804893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-sys\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.805511 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.805483    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 17:36:28.805933 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.805795    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c7fpk\""
Jul 24 17:36:28.806752 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.806363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9gdx"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.807622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-host\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.807670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-registration-dir\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.807698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-device-dir\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.807728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-sys-fs\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.807764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksq9n\" (UniqueName: \"kubernetes.io/projected/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-kube-api-access-ksq9n\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.807790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-etc-openvswitch\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.807818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff351344-a91a-46df-90b2-df47977b29c8-ovnkube-config\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.807843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-sysctl-conf\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.807871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff351344-a91a-46df-90b2-df47977b29c8-ovn-node-metrics-cert\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.807928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7df3a483-f195-40a9-b1d5-b13f47f41dc3-os-release\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.807957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ded3154-7d72-4b82-860a-35f2442eec50-etc-tuned\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.807983    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e72c8a23-8926-4a30-a022-178ed7196e5d-agent-certs\") pod \"konnectivity-agent-8p2dj\" (UID: \"e72c8a23-8926-4a30-a022-178ed7196e5d\") " pod="kube-system/konnectivity-agent-8p2dj"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.808070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-run-openvswitch\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.808100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-cni-netd\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.808127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7df3a483-f195-40a9-b1d5-b13f47f41dc3-system-cni-dir\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.808285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.808170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7df3a483-f195-40a9-b1d5-b13f47f41dc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.809038 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.808228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7df3a483-f195-40a9-b1d5-b13f47f41dc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.809038 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.808278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdrns\" (UniqueName: \"kubernetes.io/projected/7df3a483-f195-40a9-b1d5-b13f47f41dc3-kube-api-access-qdrns\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.809038 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.808310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-modprobe-d\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.809038 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.808357    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-lib-modules\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.809038 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.808370    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 17:36:28.809038 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.808385    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-var-lib-kubelet\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.809038 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4lm\" (UniqueName: \"kubernetes.io/projected/5ded3154-7d72-4b82-860a-35f2442eec50-kube-api-access-8c4lm\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.809294 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.809294 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-socket-dir\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.809294 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-systemd-units\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.809294 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-cni-bin\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.809294 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q76h7\" (UniqueName: \"kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7\") pod \"network-check-target-58pqw\" (UID: \"c8818603-e888-4c73-9e69-578a936067ae\") " pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:28.809294 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-sysconfig\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.809294 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-systemd\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.809294 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-kubelet\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.809294 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff351344-a91a-46df-90b2-df47977b29c8-ovnkube-script-lib\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tp47\" (UniqueName: \"kubernetes.io/projected/ff351344-a91a-46df-90b2-df47977b29c8-kube-api-access-6tp47\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7df3a483-f195-40a9-b1d5-b13f47f41dc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ded3154-7d72-4b82-860a-35f2442eec50-tmp\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-etc-selinux\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809429    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7df3a483-f195-40a9-b1d5-b13f47f41dc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-run\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-slash\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-run-netns\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-run-systemd\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-var-lib-openvswitch\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-log-socket\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff351344-a91a-46df-90b2-df47977b29c8-env-overrides\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-kubernetes\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.809702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-sysctl-d\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.810198 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809708    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e72c8a23-8926-4a30-a022-178ed7196e5d-konnectivity-ca\") pod \"konnectivity-agent-8p2dj\" (UID: \"e72c8a23-8926-4a30-a022-178ed7196e5d\") " pod="kube-system/konnectivity-agent-8p2dj"
Jul 24 17:36:28.810198 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.809752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-run-ovn\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.810477 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.810459    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:28.810595 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:28.810577    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:28.811952 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.811933    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dcwnb\""
Jul 24 17:36:28.812349 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.812328    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 17:36:28.812459 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.812418    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 17:36:28.844998 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.844975    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gm9mc"
Jul 24 17:36:28.854613 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.854594    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gm9mc"
Jul 24 17:36:28.892262 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.892238    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 17:36:28.910371 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7df3a483-f195-40a9-b1d5-b13f47f41dc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.910486 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-os-release\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.910486 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-cni-binary-copy\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.910486 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-multus-conf-dir\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.910670 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttdtf\" (UniqueName: \"kubernetes.io/projected/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-kube-api-access-ttdtf\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.910670 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ded3154-7d72-4b82-860a-35f2442eec50-tmp\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.910670 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-run-k8s-cni-cncf-io\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.910828 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3724c750-78f9-43b2-bf42-eb3d0d7a6c4b-serviceca\") pod \"node-ca-gnvrc\" (UID: \"3724c750-78f9-43b2-bf42-eb3d0d7a6c4b\") " pod="openshift-image-registry/node-ca-gnvrc"
Jul 24 17:36:28.910828 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7df3a483-f195-40a9-b1d5-b13f47f41dc3-os-release\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.910828 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-run\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.910828 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-slash\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.910828 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-run-netns\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.910828 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-run-systemd\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-var-lib-openvswitch\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-log-socket\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7df3a483-f195-40a9-b1d5-b13f47f41dc3-os-release\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-slash\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910923    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910933    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-run-ovn\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-var-lib-openvswitch\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-run-ovn\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910958    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-run\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-node-log\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-run-systemd\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910978    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-run-netns\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910976    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-log-socket\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.910995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7df3a483-f195-40a9-b1d5-b13f47f41dc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-node-log\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-run-netns\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.911100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e79a4ac8-d313-44f1-a654-ef419b6a1109-dbus\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-sys\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-host\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-device-dir\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-sys\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-sys-fs\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-host\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-device-dir\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksq9n\" (UniqueName: \"kubernetes.io/projected/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-kube-api-access-ksq9n\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-sys-fs\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff351344-a91a-46df-90b2-df47977b29c8-ovnkube-config\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff351344-a91a-46df-90b2-df47977b29c8-ovn-node-metrics-cert\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-multus-socket-dir-parent\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.911900 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911888    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c47e251f-f0c5-48d9-80bd-b09ad0bcea70-hosts-file\") pod \"node-resolver-r9gdx\" (UID: \"c47e251f-f0c5-48d9-80bd-b09ad0bcea70\") " pod="openshift-dns/node-resolver-r9gdx"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e79a4ac8-d313-44f1-a654-ef419b6a1109-kubelet-config\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ded3154-7d72-4b82-860a-35f2442eec50-etc-tuned\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-cni-netd\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff351344-a91a-46df-90b2-df47977b29c8-ovnkube-config\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.911973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7df3a483-f195-40a9-b1d5-b13f47f41dc3-system-cni-dir\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7df3a483-f195-40a9-b1d5-b13f47f41dc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-cni-netd\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-var-lib-cni-bin\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-var-lib-cni-multus\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-run-multus-certs\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7df3a483-f195-40a9-b1d5-b13f47f41dc3-system-cni-dir\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47e251f-f0c5-48d9-80bd-b09ad0bcea70-tmp-dir\") pod \"node-resolver-r9gdx\" (UID: \"c47e251f-f0c5-48d9-80bd-b09ad0bcea70\") " pod="openshift-dns/node-resolver-r9gdx"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-system-cni-dir\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-modprobe-d\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-lib-modules\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7df3a483-f195-40a9-b1d5-b13f47f41dc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.912514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-var-lib-kubelet\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-var-lib-kubelet\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-socket-dir\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-etc-kubernetes\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-lib-modules\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8aaee88-8c0c-4d42-88d2-521004be9f53-host-slash\") pod \"iptables-alerter-9l4cq\" (UID: \"a8aaee88-8c0c-4d42-88d2-521004be9f53\") " pod="openshift-network-operator/iptables-alerter-9l4cq"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-modprobe-d\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76h7\" (UniqueName: \"kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7\") pod \"network-check-target-58pqw\" (UID: \"c8818603-e888-4c73-9e69-578a936067ae\") " pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-sysconfig\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-socket-dir\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-systemd\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-sysconfig\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-hostroot\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-systemd\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.913276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-etc-selinux\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7df3a483-f195-40a9-b1d5-b13f47f41dc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-etc-selinux\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-cnibin\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7cc4\" (UniqueName: \"kubernetes.io/projected/3724c750-78f9-43b2-bf42-eb3d0d7a6c4b-kube-api-access-w7cc4\") pod \"node-ca-gnvrc\" (UID: \"3724c750-78f9-43b2-bf42-eb3d0d7a6c4b\") " pod="openshift-image-registry/node-ca-gnvrc"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7df3a483-f195-40a9-b1d5-b13f47f41dc3-cnibin\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff351344-a91a-46df-90b2-df47977b29c8-env-overrides\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-kubernetes\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-sysctl-d\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7df3a483-f195-40a9-b1d5-b13f47f41dc3-cnibin\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e72c8a23-8926-4a30-a022-178ed7196e5d-konnectivity-ca\") pod \"konnectivity-agent-8p2dj\" (UID: \"e72c8a23-8926-4a30-a022-178ed7196e5d\") " pod="kube-system/konnectivity-agent-8p2dj"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7df3a483-f195-40a9-b1d5-b13f47f41dc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-kubernetes\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-multus-daemon-config\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-registration-dir\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.916510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-etc-openvswitch\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8782\" (UniqueName: \"kubernetes.io/projected/c47e251f-f0c5-48d9-80bd-b09ad0bcea70-kube-api-access-n8782\") pod \"node-resolver-r9gdx\" (UID: \"c47e251f-f0c5-48d9-80bd-b09ad0bcea70\") " pod="openshift-dns/node-resolver-r9gdx"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3724c750-78f9-43b2-bf42-eb3d0d7a6c4b-host\") pod \"node-ca-gnvrc\" (UID: \"3724c750-78f9-43b2-bf42-eb3d0d7a6c4b\") " pod="openshift-image-registry/node-ca-gnvrc"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-sysctl-conf\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e72c8a23-8926-4a30-a022-178ed7196e5d-agent-certs\") pod \"konnectivity-agent-8p2dj\" (UID: \"e72c8a23-8926-4a30-a022-178ed7196e5d\") " pod="kube-system/konnectivity-agent-8p2dj"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-run-openvswitch\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrns\" (UniqueName: \"kubernetes.io/projected/7df3a483-f195-40a9-b1d5-b13f47f41dc3-kube-api-access-qdrns\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7df3a483-f195-40a9-b1d5-b13f47f41dc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.912962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-multus-cni-dir\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913014    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8aaee88-8c0c-4d42-88d2-521004be9f53-iptables-alerter-script\") pod \"iptables-alerter-9l4cq\" (UID: \"a8aaee88-8c0c-4d42-88d2-521004be9f53\") " pod="openshift-network-operator/iptables-alerter-9l4cq"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcm66\" (UniqueName: \"kubernetes.io/projected/e2551d48-cd18-4944-b555-c595868e3881-kube-api-access-kcm66\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-sysctl-d\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4lm\" (UniqueName: \"kubernetes.io/projected/5ded3154-7d72-4b82-860a-35f2442eec50-kube-api-access-8c4lm\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff351344-a91a-46df-90b2-df47977b29c8-env-overrides\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-systemd-units\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-cni-bin\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ded3154-7d72-4b82-860a-35f2442eec50-etc-sysctl-conf\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.917395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tp47\" (UniqueName: \"kubernetes.io/projected/ff351344-a91a-46df-90b2-df47977b29c8-kube-api-access-6tp47\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-var-lib-kubelet\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sj5q\" (UniqueName: \"kubernetes.io/projected/a8aaee88-8c0c-4d42-88d2-521004be9f53-kube-api-access-7sj5q\") pod \"iptables-alerter-9l4cq\" (UID: \"a8aaee88-8c0c-4d42-88d2-521004be9f53\") " pod="openshift-network-operator/iptables-alerter-9l4cq"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913480    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7df3a483-f195-40a9-b1d5-b13f47f41dc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913480    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e72c8a23-8926-4a30-a022-178ed7196e5d-konnectivity-ca\") pod \"konnectivity-agent-8p2dj\" (UID: \"e72c8a23-8926-4a30-a022-178ed7196e5d\") " pod="kube-system/konnectivity-agent-8p2dj"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-kubelet\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-cni-bin\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff351344-a91a-46df-90b2-df47977b29c8-ovnkube-script-lib\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-host-kubelet\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913645    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-run-openvswitch\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-etc-openvswitch\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-registration-dir\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.913787    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff351344-a91a-46df-90b2-df47977b29c8-systemd-units\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.914078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff351344-a91a-46df-90b2-df47977b29c8-ovnkube-script-lib\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.914283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ded3154-7d72-4b82-860a-35f2442eec50-tmp\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.915667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff351344-a91a-46df-90b2-df47977b29c8-ovn-node-metrics-cert\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.918154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.915819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ded3154-7d72-4b82-860a-35f2442eec50-etc-tuned\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.919141 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.916355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e72c8a23-8926-4a30-a022-178ed7196e5d-agent-certs\") pod \"konnectivity-agent-8p2dj\" (UID: \"e72c8a23-8926-4a30-a022-178ed7196e5d\") " pod="kube-system/konnectivity-agent-8p2dj"
Jul 24 17:36:28.927896 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:28.927871    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:36:28.927989 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:28.927903    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:36:28.927989 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:28.927919    2581 projected.go:196] Error preparing data for projected volume kube-api-access-q76h7 for pod openshift-network-diagnostics/network-check-target-58pqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:36:28.927989 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:28.927982    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7 podName:c8818603-e888-4c73-9e69-578a936067ae nodeName:}" failed. No retries permitted until 2026-07-24 17:36:29.427959922 +0000 UTC m=+2.944470951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q76h7" (UniqueName: "kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7") pod "network-check-target-58pqw" (UID: "c8818603-e888-4c73-9e69-578a936067ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:36:28.929857 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.929832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4lm\" (UniqueName: \"kubernetes.io/projected/5ded3154-7d72-4b82-860a-35f2442eec50-kube-api-access-8c4lm\") pod \"tuned-rddq7\" (UID: \"5ded3154-7d72-4b82-860a-35f2442eec50\") " pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:28.930717 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.930694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksq9n\" (UniqueName: \"kubernetes.io/projected/3b5846ea-0cef-4a61-9aeb-05cefbc1d909-kube-api-access-ksq9n\") pod \"aws-ebs-csi-driver-node-xwsr8\" (UID: \"3b5846ea-0cef-4a61-9aeb-05cefbc1d909\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:28.932155 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.932128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdrns\" (UniqueName: \"kubernetes.io/projected/7df3a483-f195-40a9-b1d5-b13f47f41dc3-kube-api-access-qdrns\") pod \"multus-additional-cni-plugins-6h2l8\" (UID: \"7df3a483-f195-40a9-b1d5-b13f47f41dc3\") " pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:28.934425 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.934403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tp47\" (UniqueName: \"kubernetes.io/projected/ff351344-a91a-46df-90b2-df47977b29c8-kube-api-access-6tp47\") pod \"ovnkube-node-t6krb\" (UID: \"ff351344-a91a-46df-90b2-df47977b29c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:28.957184 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.957129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal" event={"ID":"fc5bec91994039b1fb9327533b8ad0c7","Type":"ContainerStarted","Data":"10984b5a7814a2d7be693fd3c8d8b9c7524ad981a47163659ce7e4ad3f56feec"}
Jul 24 17:36:28.958238 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:28.958214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal" event={"ID":"780251891b48cdf9ed19946decb1fe94","Type":"ContainerStarted","Data":"0554d492a0c8c89c2d1b8e6b5157d3e03f0421ff5102636d59b924197d74467e"}
Jul 24 17:36:29.014832 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.014758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:29.014832 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.014795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7cc4\" (UniqueName: \"kubernetes.io/projected/3724c750-78f9-43b2-bf42-eb3d0d7a6c4b-kube-api-access-w7cc4\") pod \"node-ca-gnvrc\" (UID: \"3724c750-78f9-43b2-bf42-eb3d0d7a6c4b\") " pod="openshift-image-registry/node-ca-gnvrc"
Jul 24 17:36:29.014832 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.014823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-multus-daemon-config\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015090 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.014848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8782\" (UniqueName: \"kubernetes.io/projected/c47e251f-f0c5-48d9-80bd-b09ad0bcea70-kube-api-access-n8782\") pod \"node-resolver-r9gdx\" (UID: \"c47e251f-f0c5-48d9-80bd-b09ad0bcea70\") " pod="openshift-dns/node-resolver-r9gdx"
Jul 24 17:36:29.015090 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.014868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3724c750-78f9-43b2-bf42-eb3d0d7a6c4b-host\") pod \"node-ca-gnvrc\" (UID: \"3724c750-78f9-43b2-bf42-eb3d0d7a6c4b\") " pod="openshift-image-registry/node-ca-gnvrc"
Jul 24 17:36:29.015090 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.014893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-multus-cni-dir\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015090 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.014913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8aaee88-8c0c-4d42-88d2-521004be9f53-iptables-alerter-script\") pod \"iptables-alerter-9l4cq\" (UID: \"a8aaee88-8c0c-4d42-88d2-521004be9f53\") " pod="openshift-network-operator/iptables-alerter-9l4cq"
Jul 24 17:36:29.015090 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.014936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcm66\" (UniqueName: \"kubernetes.io/projected/e2551d48-cd18-4944-b555-c595868e3881-kube-api-access-kcm66\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:29.015090 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.014960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-var-lib-kubelet\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015090 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.014982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sj5q\" (UniqueName: \"kubernetes.io/projected/a8aaee88-8c0c-4d42-88d2-521004be9f53-kube-api-access-7sj5q\") pod \"iptables-alerter-9l4cq\" (UID: \"a8aaee88-8c0c-4d42-88d2-521004be9f53\") " pod="openshift-network-operator/iptables-alerter-9l4cq"
Jul 24 17:36:29.015090 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:29.015090 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-os-release\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015107    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-os-release\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015160    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3724c750-78f9-43b2-bf42-eb3d0d7a6c4b-host\") pod \"node-ca-gnvrc\" (UID: \"3724c750-78f9-43b2-bf42-eb3d0d7a6c4b\") " pod="openshift-image-registry/node-ca-gnvrc"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-cni-binary-copy\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-multus-cni-dir\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-multus-conf-dir\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttdtf\" (UniqueName: \"kubernetes.io/projected/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-kube-api-access-ttdtf\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-multus-conf-dir\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-run-k8s-cni-cncf-io\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3724c750-78f9-43b2-bf42-eb3d0d7a6c4b-serviceca\") pod \"node-ca-gnvrc\" (UID: \"3724c750-78f9-43b2-bf42-eb3d0d7a6c4b\") " pod="openshift-image-registry/node-ca-gnvrc"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-run-netns\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e79a4ac8-d313-44f1-a654-ef419b6a1109-dbus\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-multus-socket-dir-parent\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.015500 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c47e251f-f0c5-48d9-80bd-b09ad0bcea70-hosts-file\") pod \"node-resolver-r9gdx\" (UID: \"c47e251f-f0c5-48d9-80bd-b09ad0bcea70\") " pod="openshift-dns/node-resolver-r9gdx"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e79a4ac8-d313-44f1-a654-ef419b6a1109-kubelet-config\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-var-lib-cni-bin\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-var-lib-cni-multus\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-run-netns\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e79a4ac8-d313-44f1-a654-ef419b6a1109-dbus\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-multus-daemon-config\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-multus-socket-dir-parent\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-run-k8s-cni-cncf-io\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-var-lib-cni-bin\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c47e251f-f0c5-48d9-80bd-b09ad0bcea70-hosts-file\") pod \"node-resolver-r9gdx\" (UID: \"c47e251f-f0c5-48d9-80bd-b09ad0bcea70\") " pod="openshift-dns/node-resolver-r9gdx"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e79a4ac8-d313-44f1-a654-ef419b6a1109-kubelet-config\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-run-multus-certs\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47e251f-f0c5-48d9-80bd-b09ad0bcea70-tmp-dir\") pod \"node-resolver-r9gdx\" (UID: \"c47e251f-f0c5-48d9-80bd-b09ad0bcea70\") " pod="openshift-dns/node-resolver-r9gdx"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-var-lib-kubelet\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.015984    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-run-multus-certs\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-cni-binary-copy\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.016138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8aaee88-8c0c-4d42-88d2-521004be9f53-iptables-alerter-script\") pod \"iptables-alerter-9l4cq\" (UID: \"a8aaee88-8c0c-4d42-88d2-521004be9f53\") " pod="openshift-network-operator/iptables-alerter-9l4cq"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:29.014914    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-system-cni-dir\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-etc-kubernetes\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:29.016238    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs podName:e2551d48-cd18-4944-b555-c595868e3881 nodeName:}" failed. No retries permitted until 2026-07-24 17:36:29.516219877 +0000 UTC m=+3.032730918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs") pod "network-metrics-daemon-6fck2" (UID: "e2551d48-cd18-4944-b555-c595868e3881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-etc-kubernetes\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8aaee88-8c0c-4d42-88d2-521004be9f53-host-slash\") pod \"iptables-alerter-9l4cq\" (UID: \"a8aaee88-8c0c-4d42-88d2-521004be9f53\") " pod="openshift-network-operator/iptables-alerter-9l4cq"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47e251f-f0c5-48d9-80bd-b09ad0bcea70-tmp-dir\") pod \"node-resolver-r9gdx\" (UID: \"c47e251f-f0c5-48d9-80bd-b09ad0bcea70\") " pod="openshift-dns/node-resolver-r9gdx"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-system-cni-dir\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8aaee88-8c0c-4d42-88d2-521004be9f53-host-slash\") pod \"iptables-alerter-9l4cq\" (UID: \"a8aaee88-8c0c-4d42-88d2-521004be9f53\") " pod="openshift-network-operator/iptables-alerter-9l4cq"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:29.016167    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-hostroot\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016352    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-hostroot\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:29.016360    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret podName:e79a4ac8-d313-44f1-a654-ef419b6a1109 nodeName:}" failed. No retries permitted until 2026-07-24 17:36:29.516349677 +0000 UTC m=+3.032860729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret") pod "global-pull-secret-syncer-zdzrg" (UID: "e79a4ac8-d313-44f1-a654-ef419b6a1109") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-cnibin\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016401    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-host-var-lib-cni-multus\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016442    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-cnibin\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.017001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.016445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3724c750-78f9-43b2-bf42-eb3d0d7a6c4b-serviceca\") pod \"node-ca-gnvrc\" (UID: \"3724c750-78f9-43b2-bf42-eb3d0d7a6c4b\") " pod="openshift-image-registry/node-ca-gnvrc"
Jul 24 17:36:29.024219 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.024192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7cc4\" (UniqueName: \"kubernetes.io/projected/3724c750-78f9-43b2-bf42-eb3d0d7a6c4b-kube-api-access-w7cc4\") pod \"node-ca-gnvrc\" (UID: \"3724c750-78f9-43b2-bf42-eb3d0d7a6c4b\") " pod="openshift-image-registry/node-ca-gnvrc"
Jul 24 17:36:29.024615 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.024594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8782\" (UniqueName: \"kubernetes.io/projected/c47e251f-f0c5-48d9-80bd-b09ad0bcea70-kube-api-access-n8782\") pod \"node-resolver-r9gdx\" (UID: \"c47e251f-f0c5-48d9-80bd-b09ad0bcea70\") " pod="openshift-dns/node-resolver-r9gdx"
Jul 24 17:36:29.025963 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.025810    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sj5q\" (UniqueName: \"kubernetes.io/projected/a8aaee88-8c0c-4d42-88d2-521004be9f53-kube-api-access-7sj5q\") pod \"iptables-alerter-9l4cq\" (UID: \"a8aaee88-8c0c-4d42-88d2-521004be9f53\") " pod="openshift-network-operator/iptables-alerter-9l4cq"
Jul 24 17:36:29.026199 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.026177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcm66\" (UniqueName: \"kubernetes.io/projected/e2551d48-cd18-4944-b555-c595868e3881-kube-api-access-kcm66\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:29.027629 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.027606    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttdtf\" (UniqueName: \"kubernetes.io/projected/44297c4d-ce4b-4b3b-b9ef-52e16621e48d-kube-api-access-ttdtf\") pod \"multus-7pvf9\" (UID: \"44297c4d-ce4b-4b3b-b9ef-52e16621e48d\") " pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.093470 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.093438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rddq7"
Jul 24 17:36:29.099862 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.099840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8"
Jul 24 17:36:29.110683 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.110661    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8p2dj"
Jul 24 17:36:29.117474 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.117452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:29.124381 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.124358    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6h2l8"
Jul 24 17:36:29.131960 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.131942    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7pvf9"
Jul 24 17:36:29.140004 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.139988    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gnvrc"
Jul 24 17:36:29.148498 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.148481    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9l4cq"
Jul 24 17:36:29.153959 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.153941    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9gdx"
Jul 24 17:36:29.520269 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.520232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76h7\" (UniqueName: \"kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7\") pod \"network-check-target-58pqw\" (UID: \"c8818603-e888-4c73-9e69-578a936067ae\") " pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:29.520269 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.520270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:29.520514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.520304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:29.520514 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:29.520401    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:36:29.520514 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:29.520406    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:36:29.520514 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:29.520420    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:36:29.520514 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:29.520437    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:36:29.520514 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:29.520449    2581 projected.go:196] Error preparing data for projected volume kube-api-access-q76h7 for pod openshift-network-diagnostics/network-check-target-58pqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:36:29.520514 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:29.520462    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret podName:e79a4ac8-d313-44f1-a654-ef419b6a1109 nodeName:}" failed. No retries permitted until 2026-07-24 17:36:30.520442135 +0000 UTC m=+4.036953181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret") pod "global-pull-secret-syncer-zdzrg" (UID: "e79a4ac8-d313-44f1-a654-ef419b6a1109") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:36:29.520514 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:29.520479    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs podName:e2551d48-cd18-4944-b555-c595868e3881 nodeName:}" failed. No retries permitted until 2026-07-24 17:36:30.520470262 +0000 UTC m=+4.036981280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs") pod "network-metrics-daemon-6fck2" (UID: "e2551d48-cd18-4944-b555-c595868e3881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:36:29.520514 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:29.520494    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7 podName:c8818603-e888-4c73-9e69-578a936067ae nodeName:}" failed. No retries permitted until 2026-07-24 17:36:30.520485799 +0000 UTC m=+4.036996813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q76h7" (UniqueName: "kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7") pod "network-check-target-58pqw" (UID: "c8818603-e888-4c73-9e69-578a936067ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:36:29.858101 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.858059    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 17:31:28 +0000 UTC" deadline="2028-03-28 04:10:10.610961747 +0000 UTC"
Jul 24 17:36:29.858101 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:29.858095    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14698h33m40.752872026s"
Jul 24 17:36:29.986311 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:29.986283    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df3a483_f195_40a9_b1d5_b13f47f41dc3.slice/crio-93ed3edf391271f037744a3476b8bf1ae81efa914a398d969291135edfe4e528 WatchSource:0}: Error finding container 93ed3edf391271f037744a3476b8bf1ae81efa914a398d969291135edfe4e528: Status 404 returned error can't find the container with id 93ed3edf391271f037744a3476b8bf1ae81efa914a398d969291135edfe4e528
Jul 24 17:36:29.989749 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:29.989684    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44297c4d_ce4b_4b3b_b9ef_52e16621e48d.slice/crio-8bddd02f2aa48d157c9ec555387423fd21e8e7b70fb0d3a1470d32140b1a58a3 WatchSource:0}: Error finding container 8bddd02f2aa48d157c9ec555387423fd21e8e7b70fb0d3a1470d32140b1a58a3: Status 404 returned error can't find the container with id 8bddd02f2aa48d157c9ec555387423fd21e8e7b70fb0d3a1470d32140b1a58a3
Jul 24 17:36:29.993208 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:29.993188    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff351344_a91a_46df_90b2_df47977b29c8.slice/crio-430a3d284c5cd7fca17c6b79dd5e71a4e9e3948d0e678c9435d2b887e94c49b5 WatchSource:0}: Error finding container 430a3d284c5cd7fca17c6b79dd5e71a4e9e3948d0e678c9435d2b887e94c49b5: Status 404 returned error can't find the container with id 430a3d284c5cd7fca17c6b79dd5e71a4e9e3948d0e678c9435d2b887e94c49b5
Jul 24 17:36:29.994860 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:29.994705    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47e251f_f0c5_48d9_80bd_b09ad0bcea70.slice/crio-ec906e2006bc5680f7b5ccee8a26a5c65e03996128df020e37b4a7ec012f735a WatchSource:0}: Error finding container ec906e2006bc5680f7b5ccee8a26a5c65e03996128df020e37b4a7ec012f735a: Status 404 returned error can't find the container with id ec906e2006bc5680f7b5ccee8a26a5c65e03996128df020e37b4a7ec012f735a
Jul 24 17:36:29.995474 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:29.995450    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3724c750_78f9_43b2_bf42_eb3d0d7a6c4b.slice/crio-8e6f8896360db14af8c4b041dc2e048618a420d549c215d8114dd1214876c14c WatchSource:0}: Error finding container 8e6f8896360db14af8c4b041dc2e048618a420d549c215d8114dd1214876c14c: Status 404 returned error can't find the container with id 8e6f8896360db14af8c4b041dc2e048618a420d549c215d8114dd1214876c14c
Jul 24 17:36:29.997582 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:29.996899    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5846ea_0cef_4a61_9aeb_05cefbc1d909.slice/crio-a227d5cca534be776a86d9389713964990aa65dc899d8094e95ec2cf92ecfc8d WatchSource:0}: Error finding container a227d5cca534be776a86d9389713964990aa65dc899d8094e95ec2cf92ecfc8d: Status 404 returned error can't find the container with id a227d5cca534be776a86d9389713964990aa65dc899d8094e95ec2cf92ecfc8d
Jul 24 17:36:29.997582 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:29.997471    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ded3154_7d72_4b82_860a_35f2442eec50.slice/crio-a00f9ca3f943c8f94b43d1bc64692e0daf8d1e10d9e17aee525710f4a5aca06e WatchSource:0}: Error finding container a00f9ca3f943c8f94b43d1bc64692e0daf8d1e10d9e17aee525710f4a5aca06e: Status 404 returned error can't find the container with id a00f9ca3f943c8f94b43d1bc64692e0daf8d1e10d9e17aee525710f4a5aca06e
Jul 24 17:36:29.999263 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:29.999246    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8aaee88_8c0c_4d42_88d2_521004be9f53.slice/crio-e200a9c65f85056cde00cc0dfbb0e0df7425c833e3bb8a7b2bad6ffb15e0e0ac WatchSource:0}: Error finding container e200a9c65f85056cde00cc0dfbb0e0df7425c833e3bb8a7b2bad6ffb15e0e0ac: Status 404 returned error can't find the container with id e200a9c65f85056cde00cc0dfbb0e0df7425c833e3bb8a7b2bad6ffb15e0e0ac
Jul 24 17:36:30.000319 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:36:30.000295    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72c8a23_8926_4a30_a022_178ed7196e5d.slice/crio-c648b1232853a4e4e6c54218c7867903d9fe8738158c4580c64abb45bf0eea35 WatchSource:0}: Error finding container c648b1232853a4e4e6c54218c7867903d9fe8738158c4580c64abb45bf0eea35: Status 404 returned error can't find the container with id c648b1232853a4e4e6c54218c7867903d9fe8738158c4580c64abb45bf0eea35
Jul 24 17:36:30.529759 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.529676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:30.529759 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.529744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76h7\" (UniqueName: \"kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7\") pod \"network-check-target-58pqw\" (UID: \"c8818603-e888-4c73-9e69-578a936067ae\") " pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:30.529959 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.529775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:30.529959 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:30.529856    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:36:30.529959 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:30.529915    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs podName:e2551d48-cd18-4944-b555-c595868e3881 nodeName:}" failed. No retries permitted until 2026-07-24 17:36:32.529897217 +0000 UTC m=+6.046408251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs") pod "network-metrics-daemon-6fck2" (UID: "e2551d48-cd18-4944-b555-c595868e3881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:36:30.529959 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:30.529854    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:36:30.529959 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:30.529960    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret podName:e79a4ac8-d313-44f1-a654-ef419b6a1109 nodeName:}" failed. No retries permitted until 2026-07-24 17:36:32.529946898 +0000 UTC m=+6.046457922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret") pod "global-pull-secret-syncer-zdzrg" (UID: "e79a4ac8-d313-44f1-a654-ef419b6a1109") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:36:30.530216 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:30.530004    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:36:30.530216 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:30.530018    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:36:30.530216 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:30.530031    2581 projected.go:196] Error preparing data for projected volume kube-api-access-q76h7 for pod openshift-network-diagnostics/network-check-target-58pqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:36:30.530216 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:30.530069    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7 podName:c8818603-e888-4c73-9e69-578a936067ae nodeName:}" failed. No retries permitted until 2026-07-24 17:36:32.530059499 +0000 UTC m=+6.046570519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q76h7" (UniqueName: "kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7") pod "network-check-target-58pqw" (UID: "c8818603-e888-4c73-9e69-578a936067ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:36:30.858491 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.858426    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 17:31:28 +0000 UTC" deadline="2028-04-16 11:08:12.930987544 +0000 UTC"
Jul 24 17:36:30.858491 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.858490    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15161h31m42.072501773s"
Jul 24 17:36:30.954734 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.954143    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:30.954734 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:30.954272    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:30.954734 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.954451    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:30.954734 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:30.954529    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:30.954734 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.954601    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:30.954734 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:30.954668    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:30.968957 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.968774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb" event={"ID":"ff351344-a91a-46df-90b2-df47977b29c8","Type":"ContainerStarted","Data":"430a3d284c5cd7fca17c6b79dd5e71a4e9e3948d0e678c9435d2b887e94c49b5"}
Jul 24 17:36:30.977435 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.976741    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal" event={"ID":"fc5bec91994039b1fb9327533b8ad0c7","Type":"ContainerStarted","Data":"d8eab090e0c743e4f1568b4b03467e18a8d4e951b0fd6d1c7f0dc872ea8aabc8"}
Jul 24 17:36:30.980058 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.979995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9gdx" event={"ID":"c47e251f-f0c5-48d9-80bd-b09ad0bcea70","Type":"ContainerStarted","Data":"ec906e2006bc5680f7b5ccee8a26a5c65e03996128df020e37b4a7ec012f735a"}
Jul 24 17:36:30.987283 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.987228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gnvrc" event={"ID":"3724c750-78f9-43b2-bf42-eb3d0d7a6c4b","Type":"ContainerStarted","Data":"8e6f8896360db14af8c4b041dc2e048618a420d549c215d8114dd1214876c14c"}
Jul 24 17:36:30.991520 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.991473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7pvf9" event={"ID":"44297c4d-ce4b-4b3b-b9ef-52e16621e48d","Type":"ContainerStarted","Data":"8bddd02f2aa48d157c9ec555387423fd21e8e7b70fb0d3a1470d32140b1a58a3"}
Jul 24 17:36:30.998736 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:30.998710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h2l8" event={"ID":"7df3a483-f195-40a9-b1d5-b13f47f41dc3","Type":"ContainerStarted","Data":"93ed3edf391271f037744a3476b8bf1ae81efa914a398d969291135edfe4e528"}
Jul 24 17:36:31.014609 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:31.014563    2581 generic.go:356] "Generic (PLEG): container finished" podID="780251891b48cdf9ed19946decb1fe94" containerID="1603276f0e03aa861303ac23edd8853af2a998c9d78147d7375255ee79813d31" exitCode=0
Jul 24 17:36:31.014912 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:31.014849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal" event={"ID":"780251891b48cdf9ed19946decb1fe94","Type":"ContainerDied","Data":"1603276f0e03aa861303ac23edd8853af2a998c9d78147d7375255ee79813d31"}
Jul 24 17:36:31.021978 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:31.021957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8p2dj" event={"ID":"e72c8a23-8926-4a30-a022-178ed7196e5d","Type":"ContainerStarted","Data":"c648b1232853a4e4e6c54218c7867903d9fe8738158c4580c64abb45bf0eea35"}
Jul 24 17:36:31.026731 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:31.026704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9l4cq" event={"ID":"a8aaee88-8c0c-4d42-88d2-521004be9f53","Type":"ContainerStarted","Data":"e200a9c65f85056cde00cc0dfbb0e0df7425c833e3bb8a7b2bad6ffb15e0e0ac"}
Jul 24 17:36:31.038644 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:31.038592    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal" podStartSLOduration=4.038555518 podStartE2EDuration="4.038555518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:36:31.003630086 +0000 UTC m=+4.520141131" watchObservedRunningTime="2026-07-24 17:36:31.038555518 +0000 UTC m=+4.555066555"
Jul 24 17:36:31.038983 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:31.038938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rddq7" event={"ID":"5ded3154-7d72-4b82-860a-35f2442eec50","Type":"ContainerStarted","Data":"a00f9ca3f943c8f94b43d1bc64692e0daf8d1e10d9e17aee525710f4a5aca06e"}
Jul 24 17:36:31.047541 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:31.047515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8" event={"ID":"3b5846ea-0cef-4a61-9aeb-05cefbc1d909","Type":"ContainerStarted","Data":"a227d5cca534be776a86d9389713964990aa65dc899d8094e95ec2cf92ecfc8d"}
Jul 24 17:36:32.076717 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:32.076635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal" event={"ID":"780251891b48cdf9ed19946decb1fe94","Type":"ContainerStarted","Data":"5e89f00e9eba4e410f6f13b6e75a8a66304d860f0c8be683b3ab58231532f6ae"}
Jul 24 17:36:32.097032 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:32.096393    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal" podStartSLOduration=5.09637528 podStartE2EDuration="5.09637528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:36:32.095715109 +0000 UTC m=+5.612226149" watchObservedRunningTime="2026-07-24 17:36:32.09637528 +0000 UTC m=+5.612886318"
Jul 24 17:36:32.550600 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:32.550332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76h7\" (UniqueName: \"kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7\") pod \"network-check-target-58pqw\" (UID: \"c8818603-e888-4c73-9e69-578a936067ae\") " pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:32.550600 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:32.550406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:32.550600 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:32.550456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:32.550874 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:32.550603    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:36:32.550874 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:32.550666    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret podName:e79a4ac8-d313-44f1-a654-ef419b6a1109 nodeName:}" failed. No retries permitted until 2026-07-24 17:36:36.550649307 +0000 UTC m=+10.067160324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret") pod "global-pull-secret-syncer-zdzrg" (UID: "e79a4ac8-d313-44f1-a654-ef419b6a1109") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:36:32.551625 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:32.551093    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:36:32.551625 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:32.551120    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:36:32.551625 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:32.551133    2581 projected.go:196] Error preparing data for projected volume kube-api-access-q76h7 for pod openshift-network-diagnostics/network-check-target-58pqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:36:32.551625 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:32.551182    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7 podName:c8818603-e888-4c73-9e69-578a936067ae nodeName:}" failed. No retries permitted until 2026-07-24 17:36:36.551167683 +0000 UTC m=+10.067678702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q76h7" (UniqueName: "kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7") pod "network-check-target-58pqw" (UID: "c8818603-e888-4c73-9e69-578a936067ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:36:32.551625 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:32.551249    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:36:32.551625 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:32.551285    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs podName:e2551d48-cd18-4944-b555-c595868e3881 nodeName:}" failed. No retries permitted until 2026-07-24 17:36:36.551273488 +0000 UTC m=+10.067784506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs") pod "network-metrics-daemon-6fck2" (UID: "e2551d48-cd18-4944-b555-c595868e3881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:36:32.952630 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:32.952033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:32.952630 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:32.952164    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:32.953090 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:32.952918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:32.953090 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:32.953041    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:32.953090 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:32.953049    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:32.953314 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:32.953148    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:34.953301 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:34.952656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:34.953301 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:34.952792    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:34.953301 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:34.953142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:34.953301 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:34.953247    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:34.954162 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:34.954002    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:34.954162 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:34.954100    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:36.583508 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:36.583446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76h7\" (UniqueName: \"kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7\") pod \"network-check-target-58pqw\" (UID: \"c8818603-e888-4c73-9e69-578a936067ae\") " pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:36.583508 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:36.583496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:36.584058 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:36.583536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:36.584058 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:36.583620    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:36:36.584058 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:36.583642    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:36:36.584058 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:36.583656    2581 projected.go:196] Error preparing data for projected volume kube-api-access-q76h7 for pod openshift-network-diagnostics/network-check-target-58pqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:36:36.584058 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:36.583684    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:36:36.584058 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:36.583714    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7 podName:c8818603-e888-4c73-9e69-578a936067ae nodeName:}" failed. No retries permitted until 2026-07-24 17:36:44.583694847 +0000 UTC m=+18.100205865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q76h7" (UniqueName: "kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7") pod "network-check-target-58pqw" (UID: "c8818603-e888-4c73-9e69-578a936067ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:36:36.584058 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:36.583734    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret podName:e79a4ac8-d313-44f1-a654-ef419b6a1109 nodeName:}" failed. No retries permitted until 2026-07-24 17:36:44.583720309 +0000 UTC m=+18.100231332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret") pod "global-pull-secret-syncer-zdzrg" (UID: "e79a4ac8-d313-44f1-a654-ef419b6a1109") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:36:36.584058 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:36.583788    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:36:36.584058 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:36.583822    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs podName:e2551d48-cd18-4944-b555-c595868e3881 nodeName:}" failed. No retries permitted until 2026-07-24 17:36:44.583810256 +0000 UTC m=+18.100321271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs") pod "network-metrics-daemon-6fck2" (UID: "e2551d48-cd18-4944-b555-c595868e3881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:36:36.953189 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:36.953110    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:36.953350 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:36.953213    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:36.953350 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:36.953246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:36.953350 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:36.953213    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:36.953494 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:36.953348    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:36.953494 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:36.953448    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:38.951910 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:38.951878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:38.952376 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:38.951878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:38.952376 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:38.952010    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:38.952376 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:38.952099    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:38.952376 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:38.951883    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:38.952376 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:38.952189    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:40.956305 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:40.956276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:40.956760 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:40.956389    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:40.956846 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:40.956795    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:40.956901 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:40.956794    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:40.956901 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:40.956883    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:40.957018 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:40.956991    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:42.953916 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:42.952067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:42.953916 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:42.952104    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:42.953916 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:42.952068    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:42.953916 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:42.952199    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:42.953916 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:42.952299    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:42.953916 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:42.952393    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:44.642390 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:44.642353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:44.642915 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:44.642410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:44.642915 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:44.642467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76h7\" (UniqueName: \"kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7\") pod \"network-check-target-58pqw\" (UID: \"c8818603-e888-4c73-9e69-578a936067ae\") " pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:44.642915 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:44.642549    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:36:44.642915 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:44.642562    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:36:44.642915 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:44.642590    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:36:44.642915 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:44.642608    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:36:44.642915 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:44.642620    2581 projected.go:196] Error preparing data for projected volume kube-api-access-q76h7 for pod openshift-network-diagnostics/network-check-target-58pqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:36:44.642915 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:44.642641    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret podName:e79a4ac8-d313-44f1-a654-ef419b6a1109 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:00.6426226 +0000 UTC m=+34.159133635 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret") pod "global-pull-secret-syncer-zdzrg" (UID: "e79a4ac8-d313-44f1-a654-ef419b6a1109") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:36:44.642915 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:44.642656    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7 podName:c8818603-e888-4c73-9e69-578a936067ae nodeName:}" failed. No retries permitted until 2026-07-24 17:37:00.642646322 +0000 UTC m=+34.159157336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q76h7" (UniqueName: "kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7") pod "network-check-target-58pqw" (UID: "c8818603-e888-4c73-9e69-578a936067ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:36:44.642915 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:44.642672    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs podName:e2551d48-cd18-4944-b555-c595868e3881 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:00.642664134 +0000 UTC m=+34.159175156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs") pod "network-metrics-daemon-6fck2" (UID: "e2551d48-cd18-4944-b555-c595868e3881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:36:44.952791 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:44.952756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:44.952963 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:44.952803    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:44.952963 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:44.952763    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:44.952963 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:44.952884    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:44.953128 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:44.952984    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:44.953128 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:44.953044    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:46.953735 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:46.953698    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:46.954159 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:46.953820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:46.954159 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:46.953856    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:46.954159 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:46.953902    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:46.954159 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:46.953945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:46.954159 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:46.954037    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:48.952755 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:48.952603    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:48.953392 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:48.952613    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:48.953392 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:48.952833    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:48.953392 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:48.952647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:48.953392 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:48.952930    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:48.953392 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:48.953015    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:49.109545 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.109510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8p2dj" event={"ID":"e72c8a23-8926-4a30-a022-178ed7196e5d","Type":"ContainerStarted","Data":"9d03994c1df61155874e116f02dbdf8d60fd6cbdb7278165d319ed2afd100527"}
Jul 24 17:36:49.111425 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.111398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rddq7" event={"ID":"5ded3154-7d72-4b82-860a-35f2442eec50","Type":"ContainerStarted","Data":"25542a45cb3ccc7c8083eb3231e98b63c7026a688c6385a3b687cfac2d69ddb1"}
Jul 24 17:36:49.113119 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.113084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8" event={"ID":"3b5846ea-0cef-4a61-9aeb-05cefbc1d909","Type":"ContainerStarted","Data":"8c9948e9205a6da0192cdf9d683653fa1c129bedd54e63bbcf8a18ac69c67524"}
Jul 24 17:36:49.116446 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.116418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb" event={"ID":"ff351344-a91a-46df-90b2-df47977b29c8","Type":"ContainerStarted","Data":"aa7c705a9717cb93b588c01f0ad91b59f319f68d631bf98ef06312ea7dd22321"}
Jul 24 17:36:49.116552 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.116523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb" event={"ID":"ff351344-a91a-46df-90b2-df47977b29c8","Type":"ContainerStarted","Data":"95ff7c113638fa95df2b2c0fb57961a273a3ff5215c2d78c04e77b3cce6e3110"}
Jul 24 17:36:49.116635 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.116560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb" event={"ID":"ff351344-a91a-46df-90b2-df47977b29c8","Type":"ContainerStarted","Data":"f65d89719ea095e4f55e2cc3500c7060a6caab505ff8fb071c42e3213cf739ca"}
Jul 24 17:36:49.116688 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.116615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb" event={"ID":"ff351344-a91a-46df-90b2-df47977b29c8","Type":"ContainerStarted","Data":"a10eb317b9b73a8aadedcafc16466c062864b11ba1364f6c7b33318dccf36a0d"}
Jul 24 17:36:49.116738 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.116684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb" event={"ID":"ff351344-a91a-46df-90b2-df47977b29c8","Type":"ContainerStarted","Data":"d8d4057257a9943f1038e0eaa0360be0854f9d82ea216f6160d6ea7aef1859b2"}
Jul 24 17:36:49.116795 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.116735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb" event={"ID":"ff351344-a91a-46df-90b2-df47977b29c8","Type":"ContainerStarted","Data":"538acdb0d90ee8233df147af3cc972eef5cf1fa2cb8810b46c2f06fd47de612a"}
Jul 24 17:36:49.117946 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.117922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9gdx" event={"ID":"c47e251f-f0c5-48d9-80bd-b09ad0bcea70","Type":"ContainerStarted","Data":"713538bed1f15eb80fb9c349d68fb3d7870a1159a332741628bcc9807024cfd4"}
Jul 24 17:36:49.119499 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.119476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gnvrc" event={"ID":"3724c750-78f9-43b2-bf42-eb3d0d7a6c4b","Type":"ContainerStarted","Data":"bb36bd38ba23099488c20fbf0ef7f629ad3852cbfd65bd5d6a3500fba260e98d"}
Jul 24 17:36:49.120995 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.120973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7pvf9" event={"ID":"44297c4d-ce4b-4b3b-b9ef-52e16621e48d","Type":"ContainerStarted","Data":"a4172b38e045b146e45b1275b843f222f53e6c67238b640ab41350ffe7607b99"}
Jul 24 17:36:49.122521 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.122498    2581 generic.go:356] "Generic (PLEG): container finished" podID="7df3a483-f195-40a9-b1d5-b13f47f41dc3" containerID="9360edb6d2cacda1818ff85159e114ca3728c831b49af80ec9e713242875e63e" exitCode=0
Jul 24 17:36:49.122628 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.122539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h2l8" event={"ID":"7df3a483-f195-40a9-b1d5-b13f47f41dc3","Type":"ContainerDied","Data":"9360edb6d2cacda1818ff85159e114ca3728c831b49af80ec9e713242875e63e"}
Jul 24 17:36:49.126983 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.126918    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8p2dj" podStartSLOduration=4.8846186110000005 podStartE2EDuration="23.12688258s" totalImagesPullingTime="18.242263969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:36:49.126836886 +0000 UTC m=+22.643347922" watchObservedRunningTime="2026-07-24 17:36:49.12688258 +0000 UTC m=+22.643393613"
Jul 24 17:36:49.172981 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.172932    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rddq7" podStartSLOduration=4.927705315 podStartE2EDuration="23.172914856s" totalImagesPullingTime="18.245209541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:36:49.146386221 +0000 UTC m=+22.662897269" watchObservedRunningTime="2026-07-24 17:36:49.172914856 +0000 UTC m=+22.689425893"
Jul 24 17:36:49.251392 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.251349    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gnvrc" podStartSLOduration=4.004281267 podStartE2EDuration="22.251332816s" totalImagesPullingTime="18.247051549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:36:49.192680222 +0000 UTC m=+22.709191257" watchObservedRunningTime="2026-07-24 17:36:49.251332816 +0000 UTC m=+22.767843842"
Jul 24 17:36:49.251672 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:49.251650    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7pvf9" podStartSLOduration=3.955625886 podStartE2EDuration="22.251643078s" totalImagesPullingTime="18.296017192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:36:49.248076116 +0000 UTC m=+22.764587151" watchObservedRunningTime="2026-07-24 17:36:49.251643078 +0000 UTC m=+22.768154113"
Jul 24 17:36:50.126468 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:50.126427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9l4cq" event={"ID":"a8aaee88-8c0c-4d42-88d2-521004be9f53","Type":"ContainerStarted","Data":"52ff46c9ee7ed4cc2c2e7cff34bfaa04fe483dfb0426cd47f2237c04c9f5613b"}
Jul 24 17:36:50.134431 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:50.134400    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 17:36:50.144869 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:50.144830    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r9gdx" podStartSLOduration=4.896642301 podStartE2EDuration="23.144815135s" totalImagesPullingTime="18.248172834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:36:49.288219867 +0000 UTC m=+22.804730902" watchObservedRunningTime="2026-07-24 17:36:50.144815135 +0000 UTC m=+23.661326168"
Jul 24 17:36:50.144999 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:50.144981    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9l4cq" podStartSLOduration=4.901312525 podStartE2EDuration="23.144977209s" totalImagesPullingTime="18.243664684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:36:50.144603205 +0000 UTC m=+23.661114242" watchObservedRunningTime="2026-07-24 17:36:50.144977209 +0000 UTC m=+23.661488298"
Jul 24 17:36:50.871093 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:50.870931    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T17:36:50.134416825Z","UUID":"e7e056ac-a532-41d5-bff4-9cc940359ed4","Handler":null,"Name":"","Endpoint":""}
Jul 24 17:36:50.872862 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:50.872837    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 17:36:50.873041 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:50.872882    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 17:36:50.952715 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:50.952682    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:50.952887 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:50.952682    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:50.952887 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:50.952809    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:50.952887 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:50.952822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:50.953054 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:50.952923    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:50.953054 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:50.953010    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:51.134238 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:51.134123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8" event={"ID":"3b5846ea-0cef-4a61-9aeb-05cefbc1d909","Type":"ContainerStarted","Data":"1a02b1f5fedb1732cbed8d2c9ee964fd56d50c6c185f8dee86d8d6914d87ccb2"}
Jul 24 17:36:51.238943 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:51.238909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8p2dj"
Jul 24 17:36:51.239523 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:51.239501    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8p2dj"
Jul 24 17:36:51.293497 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:51.293478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8p2dj"
Jul 24 17:36:51.293869 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:51.293853    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8p2dj"
Jul 24 17:36:52.138193 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:52.137964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8" event={"ID":"3b5846ea-0cef-4a61-9aeb-05cefbc1d909","Type":"ContainerStarted","Data":"a23f2d4cba66f4f5aa74dc10f09fe0f1cdc6578fe686eab70e5d4dbc23451ead"}
Jul 24 17:36:52.141077 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:52.141055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb" event={"ID":"ff351344-a91a-46df-90b2-df47977b29c8","Type":"ContainerStarted","Data":"1d02d3db9b2a1ad1cb8ea226bcf0fad24c60be5d108f56de393172fd1fe52a3e"}
Jul 24 17:36:52.952850 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:52.952818    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:52.953059 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:52.952818    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:52.953059 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:52.952944    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:52.953059 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:52.952937    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:52.953059 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:52.953050    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:52.953255 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:52.953137    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:54.952301 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:54.952143    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:54.953003 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:54.952165    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:54.953003 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:54.952379    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:54.953003 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:54.952468    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:54.953003 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:54.952198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:54.953003 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:54.952551    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:55.151794 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:55.151760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb" event={"ID":"ff351344-a91a-46df-90b2-df47977b29c8","Type":"ContainerStarted","Data":"347c5693425c29da30bbc592a291baf6d1e9a6b5dda6c0a5aee075204598f316"}
Jul 24 17:36:55.152131 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:55.152097    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:55.152261 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:55.152190    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:55.152261 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:55.152243    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:55.153748 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:55.153721    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h2l8" event={"ID":"7df3a483-f195-40a9-b1d5-b13f47f41dc3","Type":"ContainerStarted","Data":"e75085515f4de87821d9b2f25bbcb689dcb8e4ca8c6fb7a80734e8c4fb956929"}
Jul 24 17:36:55.167512 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:55.167492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:55.167667 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:55.167653    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:36:55.193137 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:55.193103    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwsr8" podStartSLOduration=8.055696969 podStartE2EDuration="29.1930907s" totalImagesPullingTime="21.137393731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:36:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:36:52.165723609 +0000 UTC m=+25.682234645" watchObservedRunningTime="2026-07-24 17:36:55.1930907 +0000 UTC m=+28.709601736"
Jul 24 17:36:55.193430 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:55.193408    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb" podStartSLOduration=10.842729328 podStartE2EDuration="29.193400642s" totalImagesPullingTime="18.350671314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:36:55.191861926 +0000 UTC m=+28.708372953" watchObservedRunningTime="2026-07-24 17:36:55.193400642 +0000 UTC m=+28.709911675"
Jul 24 17:36:56.158149 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:56.158116    2581 generic.go:356] "Generic (PLEG): container finished" podID="7df3a483-f195-40a9-b1d5-b13f47f41dc3" containerID="e75085515f4de87821d9b2f25bbcb689dcb8e4ca8c6fb7a80734e8c4fb956929" exitCode=0
Jul 24 17:36:56.158542 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:56.158206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h2l8" event={"ID":"7df3a483-f195-40a9-b1d5-b13f47f41dc3","Type":"ContainerDied","Data":"e75085515f4de87821d9b2f25bbcb689dcb8e4ca8c6fb7a80734e8c4fb956929"}
Jul 24 17:36:56.955718 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:56.955697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:56.955801 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:56.955697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:56.955862 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:56.955799    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:56.955914 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:56.955881    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:56.955914 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:56.955697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:56.956007 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:56.955975    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:56.998590 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:56.998550    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6fck2"]
Jul 24 17:36:57.008668 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:57.008644    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-58pqw"]
Jul 24 17:36:57.010395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:57.010372    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zdzrg"]
Jul 24 17:36:57.164940 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:57.164905    2581 generic.go:356] "Generic (PLEG): container finished" podID="7df3a483-f195-40a9-b1d5-b13f47f41dc3" containerID="391946c2449764fcdfadd6a204337626cd81c49a127a4b411ab952e979f7fb22" exitCode=0
Jul 24 17:36:57.165345 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:57.165007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h2l8" event={"ID":"7df3a483-f195-40a9-b1d5-b13f47f41dc3","Type":"ContainerDied","Data":"391946c2449764fcdfadd6a204337626cd81c49a127a4b411ab952e979f7fb22"}
Jul 24 17:36:57.165345 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:57.165039    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:57.165345 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:57.165070    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:57.165345 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:57.165201    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:57.165345 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:57.165233    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:57.165729 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:57.165691    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:57.165841 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:57.165748    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:58.170356 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:58.170195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h2l8" event={"ID":"7df3a483-f195-40a9-b1d5-b13f47f41dc3","Type":"ContainerStarted","Data":"395a98a20231a9274539aba3a607022f6485cd3f33b75752b3ef57efe82ff612"}
Jul 24 17:36:58.951839 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:58.951811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:36:58.951964 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:58.951904    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:36:58.951964 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:58.951811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:36:58.952025 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:58.952011    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:36:58.952072 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:58.952048    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:36:58.952116 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:36:58.952104    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:36:59.174074 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:59.173988    2581 generic.go:356] "Generic (PLEG): container finished" podID="7df3a483-f195-40a9-b1d5-b13f47f41dc3" containerID="395a98a20231a9274539aba3a607022f6485cd3f33b75752b3ef57efe82ff612" exitCode=0
Jul 24 17:36:59.174074 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:36:59.174025    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h2l8" event={"ID":"7df3a483-f195-40a9-b1d5-b13f47f41dc3","Type":"ContainerDied","Data":"395a98a20231a9274539aba3a607022f6485cd3f33b75752b3ef57efe82ff612"}
Jul 24 17:37:00.665958 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:00.665923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76h7\" (UniqueName: \"kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7\") pod \"network-check-target-58pqw\" (UID: \"c8818603-e888-4c73-9e69-578a936067ae\") " pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:37:00.666370 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:00.665975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:37:00.666370 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:00.666096    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:37:00.666370 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:00.666120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:37:00.666370 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:00.666131    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:37:00.666370 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:00.666154    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs podName:e2551d48-cd18-4944-b555-c595868e3881 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:32.666134516 +0000 UTC m=+66.182645543 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs") pod "network-metrics-daemon-6fck2" (UID: "e2551d48-cd18-4944-b555-c595868e3881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:37:00.666370 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:00.666163    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:37:00.666370 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:00.666177    2581 projected.go:196] Error preparing data for projected volume kube-api-access-q76h7 for pod openshift-network-diagnostics/network-check-target-58pqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:37:00.666370 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:00.666227    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7 podName:c8818603-e888-4c73-9e69-578a936067ae nodeName:}" failed. No retries permitted until 2026-07-24 17:37:32.666210306 +0000 UTC m=+66.182721335 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q76h7" (UniqueName: "kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7") pod "network-check-target-58pqw" (UID: "c8818603-e888-4c73-9e69-578a936067ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:37:00.666370 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:00.666267    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:37:00.666370 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:00.666322    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret podName:e79a4ac8-d313-44f1-a654-ef419b6a1109 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:32.666304635 +0000 UTC m=+66.182815661 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret") pod "global-pull-secret-syncer-zdzrg" (UID: "e79a4ac8-d313-44f1-a654-ef419b6a1109") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:37:00.952970 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:00.952896    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:37:00.952970 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:00.952898    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:37:00.953165 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:00.953099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:37:00.953165 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:00.953121    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:37:00.953275 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:00.953159    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdzrg" podUID="e79a4ac8-d313-44f1-a654-ef419b6a1109"
Jul 24 17:37:00.953275 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:00.953217    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58pqw" podUID="c8818603-e888-4c73-9e69-578a936067ae"
Jul 24 17:37:01.848217 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.848144    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeReady"
Jul 24 17:37:01.848766 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.848277    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 17:37:01.886542 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.886513    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f77f7865d-vjglq"]
Jul 24 17:37:01.907682 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.907654    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-v8577"]
Jul 24 17:37:01.907918 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.907745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:01.911181 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.911121    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 17:37:01.911648 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.911361    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 17:37:01.911648 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.911469    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4k4kx\""
Jul 24 17:37:01.911648 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.911612    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 17:37:01.922326 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.921966    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 17:37:01.923727 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.923706    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv"]
Jul 24 17:37:01.923860 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.923838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:37:01.926276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.926193    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 24 17:37:01.926276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.926241    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 24 17:37:01.926610 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.926592    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-h6sbw\""
Jul 24 17:37:01.941291 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.941261    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"]
Jul 24 17:37:01.941410 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.941388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv"
Jul 24 17:37:01.944433 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.944409    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 17:37:01.944558 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.944418    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 17:37:01.944558 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.944539    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 17:37:01.944706 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.944605    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-zghm7\""
Jul 24 17:37:01.944893 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.944872    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 24 17:37:01.961719 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.961687    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"]
Jul 24 17:37:01.961821 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.961797    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:01.964363 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.964344    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 24 17:37:01.983766 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.983716    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv"]
Jul 24 17:37:01.983766 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.983740    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f77f7865d-vjglq"]
Jul 24 17:37:01.983766 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.983759    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-v8577"]
Jul 24 17:37:01.983958 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.983772    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"]
Jul 24 17:37:01.983958 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.983794    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2tb94"]
Jul 24 17:37:01.983958 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.983850    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:01.986557 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.986535    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 24 17:37:01.986731 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.986701    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 24 17:37:01.986814 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.986749    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 24 17:37:01.987036 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:01.987018    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 24 17:37:02.004199 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.004179    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"]
Jul 24 17:37:02.004318 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.004213    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cxc52"]
Jul 24 17:37:02.004318 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.004308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2tb94"
Jul 24 17:37:02.007777 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.007758    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 17:37:02.007886 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.007873    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 17:37:02.007886 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.007878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 17:37:02.008022 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.008007    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qpkwg\""
Jul 24 17:37:02.020251 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.020234    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2tb94"]
Jul 24 17:37:02.020351 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.020260    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cxc52"]
Jul 24 17:37:02.020411 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.020351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.022633 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.022611    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 17:37:02.022633 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.022624    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 17:37:02.023225 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.023210    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7mn5h\""
Jul 24 17:37:02.074848 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.074826    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-certificates\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.074988 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.074867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f39df8d3-ff04-4e26-afe5-a3583037cf0f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.074988 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.074921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f39df8d3-ff04-4e26-afe5-a3583037cf0f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.074988 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.074946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twv5q\" (UniqueName: \"kubernetes.io/projected/f39df8d3-ff04-4e26-afe5-a3583037cf0f-kube-api-access-twv5q\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.074988 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.074979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd108987-7a83-4e8e-bd7a-3aecc3385d87-image-registry-private-configuration\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.075143 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.074996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.075143 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd108987-7a83-4e8e-bd7a-3aecc3385d87-ca-trust-extracted\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.075143 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1e41958-e34b-45d6-bce4-d6026af4380e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-586c747fc8-wvmsv\" (UID: \"c1e41958-e34b-45d6-bce4-d6026af4380e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv"
Jul 24 17:37:02.075143 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-bound-sa-token\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.075143 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66lpz\" (UniqueName: \"kubernetes.io/projected/c1e41958-e34b-45d6-bce4-d6026af4380e-kube-api-access-66lpz\") pod \"managed-serviceaccount-addon-agent-586c747fc8-wvmsv\" (UID: \"c1e41958-e34b-45d6-bce4-d6026af4380e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv"
Jul 24 17:37:02.075143 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f39df8d3-ff04-4e26-afe5-a3583037cf0f-hub\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.075405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f39df8d3-ff04-4e26-afe5-a3583037cf0f-ca\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.075405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/76ced486-f7d7-4c74-8280-852c37e6b2ae-klusterlet-config\") pod \"klusterlet-addon-workmgr-658dff84cb-wz4l9\" (UID: \"76ced486-f7d7-4c74-8280-852c37e6b2ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:02.075405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwl6p\" (UniqueName: \"kubernetes.io/projected/76ced486-f7d7-4c74-8280-852c37e6b2ae-kube-api-access-lwl6p\") pod \"klusterlet-addon-workmgr-658dff84cb-wz4l9\" (UID: \"76ced486-f7d7-4c74-8280-852c37e6b2ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:02.075405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd108987-7a83-4e8e-bd7a-3aecc3385d87-trusted-ca\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.075405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39df8d3-ff04-4e26-afe5-a3583037cf0f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.075405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd108987-7a83-4e8e-bd7a-3aecc3385d87-installation-pull-secrets\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.075405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw69c\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-kube-api-access-jw69c\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.075405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76ced486-f7d7-4c74-8280-852c37e6b2ae-tmp\") pod \"klusterlet-addon-workmgr-658dff84cb-wz4l9\" (UID: \"76ced486-f7d7-4c74-8280-852c37e6b2ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:02.075405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d497c07a-1396-47b5-9c37-0d91fc12c7cf-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:37:02.075405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.075371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:37:02.117627 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.117597    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:02.175767 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.175738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd108987-7a83-4e8e-bd7a-3aecc3385d87-installation-pull-secrets\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.175909 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.175775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:37:02.175909 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.175796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-certificates\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.175909 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.175816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7-cert\") pod \"ingress-canary-2tb94\" (UID: \"c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7\") " pod="openshift-ingress-canary/ingress-canary-2tb94"
Jul 24 17:37:02.175909 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.175840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f39df8d3-ff04-4e26-afe5-a3583037cf0f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.175909 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.175865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f39df8d3-ff04-4e26-afe5-a3583037cf0f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.175909 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.175896    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 17:37:02.176181 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.175915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twv5q\" (UniqueName: \"kubernetes.io/projected/f39df8d3-ff04-4e26-afe5-a3583037cf0f-kube-api-access-twv5q\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.176181 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.175939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.176181 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.175955    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert podName:d497c07a-1396-47b5-9c37-0d91fc12c7cf nodeName:}" failed. No retries permitted until 2026-07-24 17:37:02.675938454 +0000 UTC m=+36.192449471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v8577" (UID: "d497c07a-1396-47b5-9c37-0d91fc12c7cf") : secret "networking-console-plugin-cert" not found
Jul 24 17:37:02.176181 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd108987-7a83-4e8e-bd7a-3aecc3385d87-image-registry-private-configuration\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.176181 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1e41958-e34b-45d6-bce4-d6026af4380e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-586c747fc8-wvmsv\" (UID: \"c1e41958-e34b-45d6-bce4-d6026af4380e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv"
Jul 24 17:37:02.176181 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-bound-sa-token\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.176181 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f39df8d3-ff04-4e26-afe5-a3583037cf0f-hub\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.176515 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f39df8d3-ff04-4e26-afe5-a3583037cf0f-ca\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.176515 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-tmp-dir\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.176515 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/76ced486-f7d7-4c74-8280-852c37e6b2ae-klusterlet-config\") pod \"klusterlet-addon-workmgr-658dff84cb-wz4l9\" (UID: \"76ced486-f7d7-4c74-8280-852c37e6b2ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:02.176515 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd108987-7a83-4e8e-bd7a-3aecc3385d87-trusted-ca\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.176515 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw69c\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-kube-api-access-jw69c\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.176515 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p7ln\" (UniqueName: \"kubernetes.io/projected/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-kube-api-access-7p7ln\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.176515 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76ced486-f7d7-4c74-8280-852c37e6b2ae-tmp\") pod \"klusterlet-addon-workmgr-658dff84cb-wz4l9\" (UID: \"76ced486-f7d7-4c74-8280-852c37e6b2ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:02.176515 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d497c07a-1396-47b5-9c37-0d91fc12c7cf-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:37:02.176515 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-certificates\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.176515 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56qm5\" (UniqueName: \"kubernetes.io/projected/c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7-kube-api-access-56qm5\") pod \"ingress-canary-2tb94\" (UID: \"c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7\") " pod="openshift-ingress-canary/ingress-canary-2tb94"
Jul 24 17:37:02.176515 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.177056 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd108987-7a83-4e8e-bd7a-3aecc3385d87-ca-trust-extracted\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.177056 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-config-volume\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.177056 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39df8d3-ff04-4e26-afe5-a3583037cf0f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.177056 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66lpz\" (UniqueName: \"kubernetes.io/projected/c1e41958-e34b-45d6-bce4-d6026af4380e-kube-api-access-66lpz\") pod \"managed-serviceaccount-addon-agent-586c747fc8-wvmsv\" (UID: \"c1e41958-e34b-45d6-bce4-d6026af4380e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv"
Jul 24 17:37:02.177056 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwl6p\" (UniqueName: \"kubernetes.io/projected/76ced486-f7d7-4c74-8280-852c37e6b2ae-kube-api-access-lwl6p\") pod \"klusterlet-addon-workmgr-658dff84cb-wz4l9\" (UID: \"76ced486-f7d7-4c74-8280-852c37e6b2ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:02.177056 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.176982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd108987-7a83-4e8e-bd7a-3aecc3385d87-ca-trust-extracted\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.177359 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.177257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76ced486-f7d7-4c74-8280-852c37e6b2ae-tmp\") pod \"klusterlet-addon-workmgr-658dff84cb-wz4l9\" (UID: \"76ced486-f7d7-4c74-8280-852c37e6b2ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:02.177359 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.177296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd108987-7a83-4e8e-bd7a-3aecc3385d87-trusted-ca\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.177973 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.177624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39df8d3-ff04-4e26-afe5-a3583037cf0f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.177973 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.177865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d497c07a-1396-47b5-9c37-0d91fc12c7cf-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:37:02.177973 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.177909    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:37:02.177973 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.177925    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f77f7865d-vjglq: secret "image-registry-tls" not found
Jul 24 17:37:02.178259 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.177982    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls podName:cd108987-7a83-4e8e-bd7a-3aecc3385d87 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:02.677963711 +0000 UTC m=+36.194474738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls") pod "image-registry-f77f7865d-vjglq" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87") : secret "image-registry-tls" not found
Jul 24 17:37:02.181402 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.181241    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd108987-7a83-4e8e-bd7a-3aecc3385d87-installation-pull-secrets\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.181402 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.181251    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f39df8d3-ff04-4e26-afe5-a3583037cf0f-ca\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.181402 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.181241    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd108987-7a83-4e8e-bd7a-3aecc3385d87-image-registry-private-configuration\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.181402 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.181351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f39df8d3-ff04-4e26-afe5-a3583037cf0f-hub\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.181402 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.181398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1e41958-e34b-45d6-bce4-d6026af4380e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-586c747fc8-wvmsv\" (UID: \"c1e41958-e34b-45d6-bce4-d6026af4380e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv"
Jul 24 17:37:02.181698 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.181438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f39df8d3-ff04-4e26-afe5-a3583037cf0f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.181698 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.181454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/76ced486-f7d7-4c74-8280-852c37e6b2ae-klusterlet-config\") pod \"klusterlet-addon-workmgr-658dff84cb-wz4l9\" (UID: \"76ced486-f7d7-4c74-8280-852c37e6b2ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:02.181772 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.181706    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f39df8d3-ff04-4e26-afe5-a3583037cf0f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.186717 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.186692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-bound-sa-token\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.186822 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.186722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw69c\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-kube-api-access-jw69c\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.187172 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.187154    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwl6p\" (UniqueName: \"kubernetes.io/projected/76ced486-f7d7-4c74-8280-852c37e6b2ae-kube-api-access-lwl6p\") pod \"klusterlet-addon-workmgr-658dff84cb-wz4l9\" (UID: \"76ced486-f7d7-4c74-8280-852c37e6b2ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:02.188895 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.188872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66lpz\" (UniqueName: \"kubernetes.io/projected/c1e41958-e34b-45d6-bce4-d6026af4380e-kube-api-access-66lpz\") pod \"managed-serviceaccount-addon-agent-586c747fc8-wvmsv\" (UID: \"c1e41958-e34b-45d6-bce4-d6026af4380e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv"
Jul 24 17:37:02.188895 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.188883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twv5q\" (UniqueName: \"kubernetes.io/projected/f39df8d3-ff04-4e26-afe5-a3583037cf0f-kube-api-access-twv5q\") pod \"cluster-proxy-proxy-agent-6bbffccb9b-sw79t\" (UID: \"f39df8d3-ff04-4e26-afe5-a3583037cf0f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.218687 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.218658    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:02.258763 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.258694    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv"
Jul 24 17:37:02.271877 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.271852    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:02.277200 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.277171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7-cert\") pod \"ingress-canary-2tb94\" (UID: \"c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7\") " pod="openshift-ingress-canary/ingress-canary-2tb94"
Jul 24 17:37:02.277312 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.277208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.277312 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.277256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-tmp-dir\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.277312 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.277292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p7ln\" (UniqueName: \"kubernetes.io/projected/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-kube-api-access-7p7ln\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.277476 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.277321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56qm5\" (UniqueName: \"kubernetes.io/projected/c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7-kube-api-access-56qm5\") pod \"ingress-canary-2tb94\" (UID: \"c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7\") " pod="openshift-ingress-canary/ingress-canary-2tb94"
Jul 24 17:37:02.277476 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.277420    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:37:02.277675 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.277498    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls podName:01447c5b-a7fc-43e5-b0f0-a0c43ee232d9 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:02.777477934 +0000 UTC m=+36.293988974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls") pod "dns-default-cxc52" (UID: "01447c5b-a7fc-43e5-b0f0-a0c43ee232d9") : secret "dns-default-metrics-tls" not found
Jul 24 17:37:02.277675 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.277547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-config-volume\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.277675 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.277596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-tmp-dir\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.278087 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.278065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-config-volume\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.280014 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.279995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7-cert\") pod \"ingress-canary-2tb94\" (UID: \"c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7\") " pod="openshift-ingress-canary/ingress-canary-2tb94"
Jul 24 17:37:02.286734 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.286679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p7ln\" (UniqueName: \"kubernetes.io/projected/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-kube-api-access-7p7ln\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.286821 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.286795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56qm5\" (UniqueName: \"kubernetes.io/projected/c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7-kube-api-access-56qm5\") pod \"ingress-canary-2tb94\" (UID: \"c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7\") " pod="openshift-ingress-canary/ingress-canary-2tb94"
Jul 24 17:37:02.293714 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.293625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:37:02.324645 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.324590    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2tb94"
Jul 24 17:37:02.496267 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.496030    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv"]
Jul 24 17:37:02.501342 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.500466    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"]
Jul 24 17:37:02.501342 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:37:02.501110    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1e41958_e34b_45d6_bce4_d6026af4380e.slice/crio-792c648b8b691b5733382aecf471fae8fef7bb48280a84d7d508027bc767b639 WatchSource:0}: Error finding container 792c648b8b691b5733382aecf471fae8fef7bb48280a84d7d508027bc767b639: Status 404 returned error can't find the container with id 792c648b8b691b5733382aecf471fae8fef7bb48280a84d7d508027bc767b639
Jul 24 17:37:02.507264 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:37:02.507232    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ced486_f7d7_4c74_8280_852c37e6b2ae.slice/crio-ddae83a33663f5e38f5e0e3dcbc9ec8694c14471f4a9376de483910dea23704f WatchSource:0}: Error finding container ddae83a33663f5e38f5e0e3dcbc9ec8694c14471f4a9376de483910dea23704f: Status 404 returned error can't find the container with id ddae83a33663f5e38f5e0e3dcbc9ec8694c14471f4a9376de483910dea23704f
Jul 24 17:37:02.538615 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.538562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"]
Jul 24 17:37:02.541719 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:37:02.541689    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf39df8d3_ff04_4e26_afe5_a3583037cf0f.slice/crio-79b9233d8b5465f18398f5c5a3de14a14a81a9f9b42ca9213569a5858576c310 WatchSource:0}: Error finding container 79b9233d8b5465f18398f5c5a3de14a14a81a9f9b42ca9213569a5858576c310: Status 404 returned error can't find the container with id 79b9233d8b5465f18398f5c5a3de14a14a81a9f9b42ca9213569a5858576c310
Jul 24 17:37:02.554673 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.554650    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2tb94"]
Jul 24 17:37:02.557702 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:37:02.557679    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f39de7_d644_4bd1_b0c2_0f1d8aab01f7.slice/crio-4e52a3cd029735fb8873e324b1111b1d5df940e8429fe01d3b242d572210aa70 WatchSource:0}: Error finding container 4e52a3cd029735fb8873e324b1111b1d5df940e8429fe01d3b242d572210aa70: Status 404 returned error can't find the container with id 4e52a3cd029735fb8873e324b1111b1d5df940e8429fe01d3b242d572210aa70
Jul 24 17:37:02.682007 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.681975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:37:02.682185 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.682088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:02.682185 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.682136    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 17:37:02.682308 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.682206    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert podName:d497c07a-1396-47b5-9c37-0d91fc12c7cf nodeName:}" failed. No retries permitted until 2026-07-24 17:37:03.682188948 +0000 UTC m=+37.198699983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v8577" (UID: "d497c07a-1396-47b5-9c37-0d91fc12c7cf") : secret "networking-console-plugin-cert" not found
Jul 24 17:37:02.682308 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.682217    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:37:02.682308 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.682232    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f77f7865d-vjglq: secret "image-registry-tls" not found
Jul 24 17:37:02.682308 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.682280    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls podName:cd108987-7a83-4e8e-bd7a-3aecc3385d87 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:03.682264345 +0000 UTC m=+37.198775387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls") pod "image-registry-f77f7865d-vjglq" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87") : secret "image-registry-tls" not found
Jul 24 17:37:02.782507 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.782429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:02.782673 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.782592    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:37:02.782673 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:02.782658    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls podName:01447c5b-a7fc-43e5-b0f0-a0c43ee232d9 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:03.782644347 +0000 UTC m=+37.299155360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls") pod "dns-default-cxc52" (UID: "01447c5b-a7fc-43e5-b0f0-a0c43ee232d9") : secret "dns-default-metrics-tls" not found
Jul 24 17:37:02.952891 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.952859    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:37:02.953504 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.952859    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:37:02.953504 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.952859    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:37:02.955842 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.955432    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 17:37:02.955842 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.955501    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lvv7h\""
Jul 24 17:37:02.955842 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.955582    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 17:37:02.955842 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.955652    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-88pnr\""
Jul 24 17:37:02.955842 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.955715    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 17:37:02.955842 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:02.955830    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 17:37:03.187105 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:03.187076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9" event={"ID":"76ced486-f7d7-4c74-8280-852c37e6b2ae","Type":"ContainerStarted","Data":"ddae83a33663f5e38f5e0e3dcbc9ec8694c14471f4a9376de483910dea23704f"}
Jul 24 17:37:03.188528 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:03.188501    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv" event={"ID":"c1e41958-e34b-45d6-bce4-d6026af4380e","Type":"ContainerStarted","Data":"792c648b8b691b5733382aecf471fae8fef7bb48280a84d7d508027bc767b639"}
Jul 24 17:37:03.190034 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:03.189994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2tb94" event={"ID":"c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7","Type":"ContainerStarted","Data":"4e52a3cd029735fb8873e324b1111b1d5df940e8429fe01d3b242d572210aa70"}
Jul 24 17:37:03.191487 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:03.191463    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t" event={"ID":"f39df8d3-ff04-4e26-afe5-a3583037cf0f","Type":"ContainerStarted","Data":"79b9233d8b5465f18398f5c5a3de14a14a81a9f9b42ca9213569a5858576c310"}
Jul 24 17:37:03.689584 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:03.689542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:03.689761 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:03.689628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:37:03.689761 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:03.689688    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:37:03.689761 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:03.689706    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f77f7865d-vjglq: secret "image-registry-tls" not found
Jul 24 17:37:03.689761 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:03.689713    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 17:37:03.689968 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:03.689773    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls podName:cd108987-7a83-4e8e-bd7a-3aecc3385d87 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:05.689753638 +0000 UTC m=+39.206264657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls") pod "image-registry-f77f7865d-vjglq" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87") : secret "image-registry-tls" not found
Jul 24 17:37:03.689968 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:03.689793    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert podName:d497c07a-1396-47b5-9c37-0d91fc12c7cf nodeName:}" failed. No retries permitted until 2026-07-24 17:37:05.689783222 +0000 UTC m=+39.206294240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v8577" (UID: "d497c07a-1396-47b5-9c37-0d91fc12c7cf") : secret "networking-console-plugin-cert" not found
Jul 24 17:37:03.790771 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:03.790735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:03.790955 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:03.790909    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:37:03.791027 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:03.791006    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls podName:01447c5b-a7fc-43e5-b0f0-a0c43ee232d9 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:05.790966943 +0000 UTC m=+39.307477980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls") pod "dns-default-cxc52" (UID: "01447c5b-a7fc-43e5-b0f0-a0c43ee232d9") : secret "dns-default-metrics-tls" not found
Jul 24 17:37:03.820286 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:03.819913    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:03.896470 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:03.896301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:04.810112 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:04.810077    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:04.866154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:04.866095    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:05.481548 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:05.481490    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:05.574193 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:05.574154    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:05.674677 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:05.674635    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:05.709757 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:05.709722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:05.709925 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:05.709770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:37:05.709925 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:05.709878    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 17:37:05.709925 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:05.709891    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:37:05.709925 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:05.709910    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f77f7865d-vjglq: secret "image-registry-tls" not found
Jul 24 17:37:05.709925 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:05.709927    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert podName:d497c07a-1396-47b5-9c37-0d91fc12c7cf nodeName:}" failed. No retries permitted until 2026-07-24 17:37:09.709914803 +0000 UTC m=+43.226425817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v8577" (UID: "d497c07a-1396-47b5-9c37-0d91fc12c7cf") : secret "networking-console-plugin-cert" not found
Jul 24 17:37:05.710127 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:05.709960    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls podName:cd108987-7a83-4e8e-bd7a-3aecc3385d87 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:09.709942681 +0000 UTC m=+43.226453698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls") pod "image-registry-f77f7865d-vjglq" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87") : secret "image-registry-tls" not found
Jul 24 17:37:05.810178 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:05.810150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:05.810546 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:05.810280    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:37:05.810546 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:05.810334    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls podName:01447c5b-a7fc-43e5-b0f0-a0c43ee232d9 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:09.81032033 +0000 UTC m=+43.326831348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls") pod "dns-default-cxc52" (UID: "01447c5b-a7fc-43e5-b0f0-a0c43ee232d9") : secret "dns-default-metrics-tls" not found
Jul 24 17:37:06.304103 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:06.303934    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:06.449855 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:06.449780    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:06.518931 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:06.518835    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:06.582387 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:06.582336    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:07.210260 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:07.210224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2tb94" event={"ID":"c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7","Type":"ContainerStarted","Data":"d5c69792983b01c7bee2b7d359899d1a43fb000524665c939361ed9dae2164e9"}
Jul 24 17:37:07.216181 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:07.216151    2581 generic.go:356] "Generic (PLEG): container finished" podID="7df3a483-f195-40a9-b1d5-b13f47f41dc3" containerID="e20550ef9f67ba36e69550b9a7613e24e1b8e41bbdb953b3faa7d2a6aef1b6e6" exitCode=0
Jul 24 17:37:07.216380 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:07.216192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h2l8" event={"ID":"7df3a483-f195-40a9-b1d5-b13f47f41dc3","Type":"ContainerDied","Data":"e20550ef9f67ba36e69550b9a7613e24e1b8e41bbdb953b3faa7d2a6aef1b6e6"}
Jul 24 17:37:07.270584 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:07.268680    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2tb94" podStartSLOduration=2.7315041349999998 podStartE2EDuration="6.268657324s" totalImagesPullingTime="3.537153189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:37:07.230080119 +0000 UTC m=+40.746591156" watchObservedRunningTime="2026-07-24 17:37:07.268657324 +0000 UTC m=+40.785168349"
Jul 24 17:37:07.788288 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:07.788199    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:08.223948 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:08.223427    2581 generic.go:356] "Generic (PLEG): container finished" podID="7df3a483-f195-40a9-b1d5-b13f47f41dc3" containerID="39388ea1ed501ff9057e0755dd895a238ad0915314a40d5e1c5150621bf1f40a" exitCode=0
Jul 24 17:37:08.223948 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:08.223507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h2l8" event={"ID":"7df3a483-f195-40a9-b1d5-b13f47f41dc3","Type":"ContainerDied","Data":"39388ea1ed501ff9057e0755dd895a238ad0915314a40d5e1c5150621bf1f40a"}
Jul 24 17:37:09.445920 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:09.445879    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:09.746674 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:09.746596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:09.746846 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:09.746762    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:37:09.746846 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:09.746775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:37:09.746938 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:09.746786    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f77f7865d-vjglq: secret "image-registry-tls" not found
Jul 24 17:37:09.746984 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:09.746965    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls podName:cd108987-7a83-4e8e-bd7a-3aecc3385d87 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:17.746945022 +0000 UTC m=+51.263456049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls") pod "image-registry-f77f7865d-vjglq" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87") : secret "image-registry-tls" not found
Jul 24 17:37:09.747045 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:09.746852    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 17:37:09.747095 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:09.747066    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert podName:d497c07a-1396-47b5-9c37-0d91fc12c7cf nodeName:}" failed. No retries permitted until 2026-07-24 17:37:17.747048052 +0000 UTC m=+51.263559079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v8577" (UID: "d497c07a-1396-47b5-9c37-0d91fc12c7cf") : secret "networking-console-plugin-cert" not found
Jul 24 17:37:09.848055 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:09.848020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:09.848199 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:09.848149    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:37:09.848264 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:09.848230    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls podName:01447c5b-a7fc-43e5-b0f0-a0c43ee232d9 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:17.8482107 +0000 UTC m=+51.364721737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls") pod "dns-default-cxc52" (UID: "01447c5b-a7fc-43e5-b0f0-a0c43ee232d9") : secret "dns-default-metrics-tls" not found
Jul 24 17:37:13.241351 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.241315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9" event={"ID":"76ced486-f7d7-4c74-8280-852c37e6b2ae","Type":"ContainerStarted","Data":"001f28737f1d6f1ee064b49da4e50c1006fc2a62fc5a22d6a12bb61c55a0d9b0"}
Jul 24 17:37:13.241796 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.241502    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:13.243296 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.243266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv" event={"ID":"c1e41958-e34b-45d6-bce4-d6026af4380e","Type":"ContainerStarted","Data":"1306f5482a32d1a452eafddd5533211959dae3a62d326fba200dab565e33dbca"}
Jul 24 17:37:13.243433 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.243333    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:37:13.245190 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.245165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t" event={"ID":"f39df8d3-ff04-4e26-afe5-a3583037cf0f","Type":"ContainerStarted","Data":"6f1849e00f43584ec7fb6891e95802b16046e7cf572bd0518117955994aef8f0"}
Jul 24 17:37:13.245299 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.245200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t" event={"ID":"f39df8d3-ff04-4e26-afe5-a3583037cf0f","Type":"ContainerStarted","Data":"496ae766fc01c738018ad463c70ef0052c9f7b96f01e19f9393d160aa5471957"}
Jul 24 17:37:13.245299 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.245213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t" event={"ID":"f39df8d3-ff04-4e26-afe5-a3583037cf0f","Type":"ContainerStarted","Data":"229043c5d44b89468bdc95c735f142b998a45f32d505cb805448063f0d74975d"}
Jul 24 17:37:13.248241 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.248223    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h2l8" event={"ID":"7df3a483-f195-40a9-b1d5-b13f47f41dc3","Type":"ContainerStarted","Data":"85d44c327f5f4ee434ca8604ab40b88a187fe13f410b6bb7081ceb313cc2546f"}
Jul 24 17:37:13.259075 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.258995    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9" podStartSLOduration=12.531578401 podStartE2EDuration="22.25898203s" totalImagesPullingTime="9.727403629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:37:13.257748446 +0000 UTC m=+46.774259481" watchObservedRunningTime="2026-07-24 17:37:13.25898203 +0000 UTC m=+46.775493066"
Jul 24 17:37:13.300589 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.300531    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6h2l8" podStartSLOduration=13.582484639 podStartE2EDuration="46.300518726s" totalImagesPullingTime="32.718034087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:36:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:37:13.299320939 +0000 UTC m=+46.815831976" watchObservedRunningTime="2026-07-24 17:37:13.300518726 +0000 UTC m=+46.817029761"
Jul 24 17:37:13.313828 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.313793    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv" podStartSLOduration=12.599329602 podStartE2EDuration="22.31378216s" totalImagesPullingTime="9.714452558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:37:13.313777543 +0000 UTC m=+46.830288578" watchObservedRunningTime="2026-07-24 17:37:13.31378216 +0000 UTC m=+46.830293193"
Jul 24 17:37:13.334405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.334361    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t" podStartSLOduration=12.659726262 podStartE2EDuration="22.334351146s" totalImagesPullingTime="9.674624884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:37:13.333629938 +0000 UTC m=+46.850140975" watchObservedRunningTime="2026-07-24 17:37:13.334351146 +0000 UTC m=+46.850862182"
Jul 24 17:37:13.428997 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.428962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:13.529488 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:13.529379    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:15.025723 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:15.025694    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:15.082208 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:15.082154    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:15.983763 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:15.983727    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:16.061753 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:16.061726    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:17.681783 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:17.681745    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:17.748402 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:17.748368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:17.806247 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:17.806220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:17.806377 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:17.806278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:37:17.806377 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:17.806370    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:37:17.806449 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:17.806386    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f77f7865d-vjglq: secret "image-registry-tls" not found
Jul 24 17:37:17.806449 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:17.806429    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 17:37:17.806513 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:17.806451    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls podName:cd108987-7a83-4e8e-bd7a-3aecc3385d87 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:33.806434432 +0000 UTC m=+67.322945451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls") pod "image-registry-f77f7865d-vjglq" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87") : secret "image-registry-tls" not found
Jul 24 17:37:17.806513 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:17.806487    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert podName:d497c07a-1396-47b5-9c37-0d91fc12c7cf nodeName:}" failed. No retries permitted until 2026-07-24 17:37:33.806473974 +0000 UTC m=+67.322984987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v8577" (UID: "d497c07a-1396-47b5-9c37-0d91fc12c7cf") : secret "networking-console-plugin-cert" not found
Jul 24 17:37:17.906868 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:17.906836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:17.907034 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:17.906950    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:37:17.907034 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:17.906999    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls podName:01447c5b-a7fc-43e5-b0f0-a0c43ee232d9 nodeName:}" failed. No retries permitted until 2026-07-24 17:37:33.906985413 +0000 UTC m=+67.423496427 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls") pod "dns-default-cxc52" (UID: "01447c5b-a7fc-43e5-b0f0-a0c43ee232d9") : secret "dns-default-metrics-tls" not found
Jul 24 17:37:19.877448 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:19.877404    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:19.978562 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:19.978509    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:21.056320 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:21.056291    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:21.108587 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:21.108545    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:21.552702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:21.552674    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:21.609870 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:21.609840    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:21.668627 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:21.668590    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:22.804799 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:22.804712    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:22.858871 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:22.858839    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:27.175636 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:27.175608    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6krb"
Jul 24 17:37:27.336163 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:27.336133    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:27.396649 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:27.396619    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:32.716865 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:32.716819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76h7\" (UniqueName: \"kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7\") pod \"network-check-target-58pqw\" (UID: \"c8818603-e888-4c73-9e69-578a936067ae\") " pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:37:32.717337 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:32.716898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:37:32.717337 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:32.716924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:37:32.719730 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:32.719710    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 17:37:32.719939 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:32.719923    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 17:37:32.720010 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:32.719993    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 17:37:32.727902 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:32.727881    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 17:37:32.727995 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:32.727943    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs podName:e2551d48-cd18-4944-b555-c595868e3881 nodeName:}" failed. No retries permitted until 2026-07-24 17:38:36.727922914 +0000 UTC m=+130.244433929 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs") pod "network-metrics-daemon-6fck2" (UID: "e2551d48-cd18-4944-b555-c595868e3881") : secret "metrics-daemon-secret" not found
Jul 24 17:37:32.729833 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:32.729810    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 17:37:32.731177 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:32.731161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e79a4ac8-d313-44f1-a654-ef419b6a1109-original-pull-secret\") pod \"global-pull-secret-syncer-zdzrg\" (UID: \"e79a4ac8-d313-44f1-a654-ef419b6a1109\") " pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:37:32.740649 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:32.740633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q76h7\" (UniqueName: \"kubernetes.io/projected/c8818603-e888-4c73-9e69-578a936067ae-kube-api-access-q76h7\") pod \"network-check-target-58pqw\" (UID: \"c8818603-e888-4c73-9e69-578a936067ae\") " pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:37:32.965724 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:32.965697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdzrg"
Jul 24 17:37:32.979132 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:32.979109    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-88pnr\""
Jul 24 17:37:32.987277 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:32.987257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:37:33.100782 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:33.100752    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zdzrg"]
Jul 24 17:37:33.114790 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:33.114775    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-58pqw"]
Jul 24 17:37:33.117660 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:37:33.117632    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8818603_e888_4c73_9e69_578a936067ae.slice/crio-414a2f6eda05736fdd6c0840bc0b44938fcbc7c14816d40be4f3c79de30d9d85 WatchSource:0}: Error finding container 414a2f6eda05736fdd6c0840bc0b44938fcbc7c14816d40be4f3c79de30d9d85: Status 404 returned error can't find the container with id 414a2f6eda05736fdd6c0840bc0b44938fcbc7c14816d40be4f3c79de30d9d85
Jul 24 17:37:33.303157 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:33.303070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-58pqw" event={"ID":"c8818603-e888-4c73-9e69-578a936067ae","Type":"ContainerStarted","Data":"414a2f6eda05736fdd6c0840bc0b44938fcbc7c14816d40be4f3c79de30d9d85"}
Jul 24 17:37:33.304069 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:33.304044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zdzrg" event={"ID":"e79a4ac8-d313-44f1-a654-ef419b6a1109","Type":"ContainerStarted","Data":"0e641b0407e30067e9e4bec29ee74029da2199f6c1da8b27cad2f943c901c4e8"}
Jul 24 17:37:33.824196 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:33.824159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:37:33.824738 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:33.824222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:37:33.824738 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:33.824303    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:37:33.824738 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:33.824322    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f77f7865d-vjglq: secret "image-registry-tls" not found
Jul 24 17:37:33.824738 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:33.824376    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 17:37:33.824738 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:33.824384    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls podName:cd108987-7a83-4e8e-bd7a-3aecc3385d87 nodeName:}" failed. No retries permitted until 2026-07-24 17:38:05.824362772 +0000 UTC m=+99.340873788 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls") pod "image-registry-f77f7865d-vjglq" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87") : secret "image-registry-tls" not found
Jul 24 17:37:33.824738 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:33.824430    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert podName:d497c07a-1396-47b5-9c37-0d91fc12c7cf nodeName:}" failed. No retries permitted until 2026-07-24 17:38:05.824413239 +0000 UTC m=+99.340924254 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v8577" (UID: "d497c07a-1396-47b5-9c37-0d91fc12c7cf") : secret "networking-console-plugin-cert" not found
Jul 24 17:37:33.925560 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:33.925519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:37:33.925731 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:33.925710    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:37:33.925798 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:37:33.925786    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls podName:01447c5b-a7fc-43e5-b0f0-a0c43ee232d9 nodeName:}" failed. No retries permitted until 2026-07-24 17:38:05.925767013 +0000 UTC m=+99.442278050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls") pod "dns-default-cxc52" (UID: "01447c5b-a7fc-43e5-b0f0-a0c43ee232d9") : secret "dns-default-metrics-tls" not found
Jul 24 17:37:38.323208 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:38.323177    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zdzrg" event={"ID":"e79a4ac8-d313-44f1-a654-ef419b6a1109","Type":"ContainerStarted","Data":"1afd0a05cc9857c7d8ddb7a141c8f293b93fdfef72f2d0601bd44312979f6608"}
Jul 24 17:37:38.324867 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:38.324829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-58pqw" event={"ID":"c8818603-e888-4c73-9e69-578a936067ae","Type":"ContainerStarted","Data":"f05f2fabf32985d43fe674e9a6a6e6df00e2a7dbc5e90925a3f873019388a9bd"}
Jul 24 17:37:38.325014 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:38.325002    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:37:38.344337 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:38.344296    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zdzrg" podStartSLOduration=66.418130657 podStartE2EDuration="1m11.344281486s" totalImagesPullingTime="4.926150829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:37:38.343399944 +0000 UTC m=+71.859910981" watchObservedRunningTime="2026-07-24 17:37:38.344281486 +0000 UTC m=+71.860792522"
Jul 24 17:37:38.362423 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:38.362377    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-58pqw" podStartSLOduration=67.800096307 podStartE2EDuration="1m12.362340128s" totalImagesPullingTime="4.562243821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:37:38.362133605 +0000 UTC m=+71.878644643" watchObservedRunningTime="2026-07-24 17:37:38.362340128 +0000 UTC m=+71.878851165"
Jul 24 17:37:41.915724 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:41.915686    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:42.004304 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:42.004268    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:42.060560 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:42.060534    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:42.569365 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:42.569321    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:42.629582 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:42.629552    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:42.688261 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:37:42.688229    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:05.864015 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:38:05.863880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:38:05.864015 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:38:05.863941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:38:05.864562 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:38:05.864028    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 17:38:05.864562 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:38:05.864041    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:38:05.864562 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:38:05.864064    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f77f7865d-vjglq: secret "image-registry-tls" not found
Jul 24 17:38:05.864562 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:38:05.864092    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert podName:d497c07a-1396-47b5-9c37-0d91fc12c7cf nodeName:}" failed. No retries permitted until 2026-07-24 17:39:09.864078157 +0000 UTC m=+163.380589173 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v8577" (UID: "d497c07a-1396-47b5-9c37-0d91fc12c7cf") : secret "networking-console-plugin-cert" not found
Jul 24 17:38:05.864562 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:38:05.864155    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls podName:cd108987-7a83-4e8e-bd7a-3aecc3385d87 nodeName:}" failed. No retries permitted until 2026-07-24 17:39:09.864138144 +0000 UTC m=+163.380649177 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls") pod "image-registry-f77f7865d-vjglq" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87") : secret "image-registry-tls" not found
Jul 24 17:38:05.964784 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:38:05.964755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:38:05.964895 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:38:05.964852    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:38:05.964934 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:38:05.964903    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls podName:01447c5b-a7fc-43e5-b0f0-a0c43ee232d9 nodeName:}" failed. No retries permitted until 2026-07-24 17:39:09.964891909 +0000 UTC m=+163.481402923 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls") pod "dns-default-cxc52" (UID: "01447c5b-a7fc-43e5-b0f0-a0c43ee232d9") : secret "dns-default-metrics-tls" not found
Jul 24 17:38:09.330803 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:38:09.330777    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-58pqw"
Jul 24 17:38:09.725398 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:38:09.725317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:09.779823 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:38:09.779786    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:19.267012 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:38:19.266980    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:36.794891 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:38:36.794843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:38:36.795353 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:38:36.794978    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 17:38:36.795353 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:38:36.795053    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs podName:e2551d48-cd18-4944-b555-c595868e3881 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:38.795037875 +0000 UTC m=+252.311548889 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs") pod "network-metrics-daemon-6fck2" (UID: "e2551d48-cd18-4944-b555-c595868e3881") : secret "metrics-daemon-secret" not found
Jul 24 17:39:04.928347 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:39:04.928292    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-f77f7865d-vjglq" podUID="cd108987-7a83-4e8e-bd7a-3aecc3385d87"
Jul 24 17:39:04.936379 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:39:04.936349    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577" podUID="d497c07a-1396-47b5-9c37-0d91fc12c7cf"
Jul 24 17:39:05.031396 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:39:05.031350    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cxc52" podUID="01447c5b-a7fc-43e5-b0f0-a0c43ee232d9"
Jul 24 17:39:05.560459 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:05.560422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:39:05.560647 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:05.560422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:39:05.971627 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:39:05.971526    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:39:09.937519 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:09.937466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:39:09.937962 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:09.937548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:39:09.937962 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:39:09.937621    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 17:39:09.937962 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:39:09.937640    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f77f7865d-vjglq: secret "image-registry-tls" not found
Jul 24 17:39:09.937962 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:39:09.937679    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 17:39:09.937962 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:39:09.937702    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls podName:cd108987-7a83-4e8e-bd7a-3aecc3385d87 nodeName:}" failed. No retries permitted until 2026-07-24 17:41:11.937686601 +0000 UTC m=+285.454197615 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls") pod "image-registry-f77f7865d-vjglq" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87") : secret "image-registry-tls" not found
Jul 24 17:39:09.937962 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:39:09.937739    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert podName:d497c07a-1396-47b5-9c37-0d91fc12c7cf nodeName:}" failed. No retries permitted until 2026-07-24 17:41:11.937724874 +0000 UTC m=+285.454235891 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v8577" (UID: "d497c07a-1396-47b5-9c37-0d91fc12c7cf") : secret "networking-console-plugin-cert" not found
Jul 24 17:39:10.038129 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:10.038078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:39:10.038347 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:39:10.038244    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:39:10.038347 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:39:10.038332    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls podName:01447c5b-a7fc-43e5-b0f0-a0c43ee232d9 nodeName:}" failed. No retries permitted until 2026-07-24 17:41:12.038310462 +0000 UTC m=+285.554821491 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls") pod "dns-default-cxc52" (UID: "01447c5b-a7fc-43e5-b0f0-a0c43ee232d9") : secret "dns-default-metrics-tls" not found
Jul 24 17:39:12.580291 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:12.580259    2581 generic.go:356] "Generic (PLEG): container finished" podID="76ced486-f7d7-4c74-8280-852c37e6b2ae" containerID="001f28737f1d6f1ee064b49da4e50c1006fc2a62fc5a22d6a12bb61c55a0d9b0" exitCode=1
Jul 24 17:39:12.580718 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:12.580330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9" event={"ID":"76ced486-f7d7-4c74-8280-852c37e6b2ae","Type":"ContainerDied","Data":"001f28737f1d6f1ee064b49da4e50c1006fc2a62fc5a22d6a12bb61c55a0d9b0"}
Jul 24 17:39:12.580718 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:12.580712    2581 scope.go:117] "RemoveContainer" containerID="001f28737f1d6f1ee064b49da4e50c1006fc2a62fc5a22d6a12bb61c55a0d9b0"
Jul 24 17:39:12.581856 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:12.581834    2581 generic.go:356] "Generic (PLEG): container finished" podID="c1e41958-e34b-45d6-bce4-d6026af4380e" containerID="1306f5482a32d1a452eafddd5533211959dae3a62d326fba200dab565e33dbca" exitCode=255
Jul 24 17:39:12.581965 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:12.581915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv" event={"ID":"c1e41958-e34b-45d6-bce4-d6026af4380e","Type":"ContainerDied","Data":"1306f5482a32d1a452eafddd5533211959dae3a62d326fba200dab565e33dbca"}
Jul 24 17:39:12.582219 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:12.582203    2581 scope.go:117] "RemoveContainer" containerID="1306f5482a32d1a452eafddd5533211959dae3a62d326fba200dab565e33dbca"
Jul 24 17:39:13.242134 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:13.242101    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:39:13.586771 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:13.586740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9" event={"ID":"76ced486-f7d7-4c74-8280-852c37e6b2ae","Type":"ContainerStarted","Data":"d596d84fc99b50f2a8965d9533591a949195d9250c9b8957303a0ee088ead44b"}
Jul 24 17:39:13.587222 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:13.586982    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:39:13.587613 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:13.587591    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:39:13.588481 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:13.588461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv" event={"ID":"c1e41958-e34b-45d6-bce4-d6026af4380e","Type":"ContainerStarted","Data":"45b240faecc72686898203896f8589c835388e33b6fe655c38b04627eac79951"}
Jul 24 17:39:16.953376 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:16.953346    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cxc52"
Jul 24 17:39:20.956046 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:39:20.955977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:40:22.076395 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:22.076366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:38.887611 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:38.887544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:40:38.887959 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:38.887677    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 17:40:38.887959 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:38.887730    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs podName:e2551d48-cd18-4944-b555-c595868e3881 nodeName:}" failed. No retries permitted until 2026-07-24 17:42:40.887715145 +0000 UTC m=+374.404226158 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs") pod "network-metrics-daemon-6fck2" (UID: "e2551d48-cd18-4944-b555-c595868e3881") : secret "metrics-daemon-secret" not found
Jul 24 17:40:45.047371 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:45.047337    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:45.148129 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:45.148100    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:47.337066 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:47.337034    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:47.394753 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:47.394726    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:48.023213 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.023181    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:48.076154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.076130    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:48.711115 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.711083    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:48.814238 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.814205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:48.850276 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.850248    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fchlk"]
Jul 24 17:40:48.854669 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.854646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.857062 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.857038    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 17:40:48.857194 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.857045    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 17:40:48.857194 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.857106    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cpj25\""
Jul 24 17:40:48.857831 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.857809    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 17:40:48.857957 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.857932    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 17:40:48.861894 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.861870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntfjc\" (UniqueName: \"kubernetes.io/projected/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-api-access-ntfjc\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.862011 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.861926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d812cd21-578e-4bb6-884c-c452e3b2c174-crio-socket\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.862011 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.861971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d812cd21-578e-4bb6-884c-c452e3b2c174-data-volume\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.862118 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.862017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.862118 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.862084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.866565 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.866545    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fchlk"]
Jul 24 17:40:48.891647 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.891589    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:48.962797 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.962732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.962797 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.962767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntfjc\" (UniqueName: \"kubernetes.io/projected/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-api-access-ntfjc\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.962988 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.962801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d812cd21-578e-4bb6-884c-c452e3b2c174-crio-socket\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.962988 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.962824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d812cd21-578e-4bb6-884c-c452e3b2c174-data-volume\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.962988 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:48.962869    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:40:48.962988 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.962881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.962988 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.962880    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d812cd21-578e-4bb6-884c-c452e3b2c174-crio-socket\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.962988 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:48.962935    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls podName:d812cd21-578e-4bb6-884c-c452e3b2c174 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:49.462919852 +0000 UTC m=+262.979430865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fchlk" (UID: "d812cd21-578e-4bb6-884c-c452e3b2c174") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:40:48.962988 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:48.962948    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:40:48.963270 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:48.963026    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm podName:d812cd21-578e-4bb6-884c-c452e3b2c174 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:49.463009025 +0000 UTC m=+262.979520043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fchlk" (UID: "d812cd21-578e-4bb6-884c-c452e3b2c174") : configmap "kube-rbac-proxy" not found
Jul 24 17:40:48.963270 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.963154    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d812cd21-578e-4bb6-884c-c452e3b2c174-data-volume\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:48.970400 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:48.970379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntfjc\" (UniqueName: \"kubernetes.io/projected/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-api-access-ntfjc\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:49.466148 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:49.466101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:49.466361 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:49.466190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:49.466361 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:49.466263    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:40:49.466361 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:49.466286    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:40:49.466361 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:49.466340    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm podName:d812cd21-578e-4bb6-884c-c452e3b2c174 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:50.466325042 +0000 UTC m=+263.982836056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fchlk" (UID: "d812cd21-578e-4bb6-884c-c452e3b2c174") : configmap "kube-rbac-proxy" not found
Jul 24 17:40:49.466361 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:49.466353    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls podName:d812cd21-578e-4bb6-884c-c452e3b2c174 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:50.46634697 +0000 UTC m=+263.982857984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fchlk" (UID: "d812cd21-578e-4bb6-884c-c452e3b2c174") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:40:49.745189 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:49.745110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:49.814425 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:49.814394    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:49.866736 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:49.866706    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:50.350990 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:50.350956    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:50.406230 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:50.406206    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:50.472163 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:50.472139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:50.472231 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:50.472212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:50.472305 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:50.472291    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:40:50.472338 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:50.472326    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:40:50.472372 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:50.472347    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls podName:d812cd21-578e-4bb6-884c-c452e3b2c174 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:52.472333904 +0000 UTC m=+265.988844917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fchlk" (UID: "d812cd21-578e-4bb6-884c-c452e3b2c174") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:40:50.472415 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:50.472371    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm podName:d812cd21-578e-4bb6-884c-c452e3b2c174 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:52.47235812 +0000 UTC m=+265.988869138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fchlk" (UID: "d812cd21-578e-4bb6-884c-c452e3b2c174") : configmap "kube-rbac-proxy" not found
Jul 24 17:40:51.031622 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:51.031592    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r9gdx_c47e251f-f0c5-48d9-80bd-b09ad0bcea70/dns-node-resolver/0.log"
Jul 24 17:40:51.216734 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:51.216700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:51.787246 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:51.787214    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:51.832697 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:51.832679    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gnvrc_3724c750-78f9-43b2-bf42-eb3d0d7a6c4b/node-ca/0.log"
Jul 24 17:40:51.997159 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:51.997124    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:52.096470 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:52.096444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:52.193779 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:52.193752    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:52.293800 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:52.293771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:52.394263 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:52.394185    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:52.441616 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:52.441547    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2tb94_c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7/serve-healthcheck-canary/0.log"
Jul 24 17:40:52.487499 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:52.487476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:52.487609 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:52.487537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:52.487654 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:52.487620    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:40:52.487688 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:52.487676    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls podName:d812cd21-578e-4bb6-884c-c452e3b2c174 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:56.487663 +0000 UTC m=+270.004174017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fchlk" (UID: "d812cd21-578e-4bb6-884c-c452e3b2c174") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:40:52.487728 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:52.487625    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:40:52.487760 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:52.487741    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm podName:d812cd21-578e-4bb6-884c-c452e3b2c174 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:56.487729419 +0000 UTC m=+270.004240434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fchlk" (UID: "d812cd21-578e-4bb6-884c-c452e3b2c174") : configmap "kube-rbac-proxy" not found
Jul 24 17:40:52.683595 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:52.683504    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:52.877970 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:52.877935    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:53.885260 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:53.885228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:54.126805 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:54.126766    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:54.286503 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:54.286421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:54.478966 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:54.478934    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:54.676865 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:54.676834    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:54.880908 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:54.880877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:55.681741 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:55.681708    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:55.782364 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:55.782341    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:55.881665 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:55.881638    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:56.288073 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:56.288040    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:56.480691 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:56.480663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:56.520889 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:56.520854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:56.521010 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:56.520929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:40:56.521010 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:56.520966    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:40:56.521077 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:56.521024    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm podName:d812cd21-578e-4bb6-884c-c452e3b2c174 nodeName:}" failed. No retries permitted until 2026-07-24 17:41:04.521009398 +0000 UTC m=+278.037520412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fchlk" (UID: "d812cd21-578e-4bb6-884c-c452e3b2c174") : configmap "kube-rbac-proxy" not found
Jul 24 17:40:56.521077 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:56.521027    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:40:56.521077 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:40:56.521065    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls podName:d812cd21-578e-4bb6-884c-c452e3b2c174 nodeName:}" failed. No retries permitted until 2026-07-24 17:41:04.52105412 +0000 UTC m=+278.037565134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fchlk" (UID: "d812cd21-578e-4bb6-884c-c452e3b2c174") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:40:56.683727 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:56.683690    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:56.879946 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:56.879912    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:57.679818 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:57.679787    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:58.280023 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:58.279994    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:58.348042 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:58.348010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:58.475689 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:58.475657    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:58.690151 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:58.690109    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:58.876686 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:40:58.876659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:00.076486 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:00.076440    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:00.284112 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:00.284079    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:00.477902 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:00.477823    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:00.677432 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:00.677405    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:00.888101 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:00.888070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:01.297510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:01.297434    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:01.680911 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:01.680884    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:01.880818 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:01.880790    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:02.080314 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:02.080291    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:02.477224 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:02.477144    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:02.731727 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:02.731654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:03.194887 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:03.194860    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:03.353457 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:03.353429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:03.461249 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:03.461175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:04.583640 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:04.583588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:41:04.584010 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:04.583672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:41:04.584100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:04.584080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d812cd21-578e-4bb6-884c-c452e3b2c174-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:41:04.585910 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:04.585892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d812cd21-578e-4bb6-884c-c452e3b2c174-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fchlk\" (UID: \"d812cd21-578e-4bb6-884c-c452e3b2c174\") " pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:41:04.763204 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:04.763172    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fchlk"
Jul 24 17:41:04.884207 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:04.884179    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fchlk"]
Jul 24 17:41:04.887143 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:41:04.887120    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd812cd21_578e_4bb6_884c_c452e3b2c174.slice/crio-2d6ff6b9178ba176537939743720e6c0928bdf135bcbce95c64681261b76640f WatchSource:0}: Error finding container 2d6ff6b9178ba176537939743720e6c0928bdf135bcbce95c64681261b76640f: Status 404 returned error can't find the container with id 2d6ff6b9178ba176537939743720e6c0928bdf135bcbce95c64681261b76640f
Jul 24 17:41:04.904002 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:04.903979    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fchlk" event={"ID":"d812cd21-578e-4bb6-884c-c452e3b2c174","Type":"ContainerStarted","Data":"2d6ff6b9178ba176537939743720e6c0928bdf135bcbce95c64681261b76640f"}
Jul 24 17:41:08.561455 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:41:08.561412    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577" podUID="d497c07a-1396-47b5-9c37-0d91fc12c7cf"
Jul 24 17:41:08.561925 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:41:08.561412    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-f77f7865d-vjglq" podUID="cd108987-7a83-4e8e-bd7a-3aecc3385d87"
Jul 24 17:41:08.916116 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:08.916039    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:41:08.916243 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:08.916049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:41:09.920392 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:09.920359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fchlk" event={"ID":"d812cd21-578e-4bb6-884c-c452e3b2c174","Type":"ContainerStarted","Data":"e0339454226f5b37ab95b9bfbe91f0a29efb99479d2563dbfbac34fc89c98222"}
Jul 24 17:41:10.945858 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:10.945816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:11.045916 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:11.045881    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:11.096277 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:11.096247    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:11.419188 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:11.419153    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:11.526941 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:11.526906    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:11.627103 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:11.627068    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:11.815980 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:11.815947    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:11.876974 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:11.876941    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:11.948671 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:11.948640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:41:11.949133 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:11.948699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:41:11.951702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:11.951677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d497c07a-1396-47b5-9c37-0d91fc12c7cf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v8577\" (UID: \"d497c07a-1396-47b5-9c37-0d91fc12c7cf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:41:11.951830 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:11.951750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"image-registry-f77f7865d-vjglq\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") " pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:41:12.040031 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.039998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:12.050011 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.049984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:41:12.052479 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.052457    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01447c5b-a7fc-43e5-b0f0-a0c43ee232d9-metrics-tls\") pod \"dns-default-cxc52\" (UID: \"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9\") " pod="openshift-dns/dns-default-cxc52"
Jul 24 17:41:12.141494 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.141409    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:12.157004 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.156978    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7mn5h\""
Jul 24 17:41:12.165183 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.165160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cxc52"
Jul 24 17:41:12.219704 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.219680    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4k4kx\""
Jul 24 17:41:12.219852 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.219750    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-h6sbw\""
Jul 24 17:41:12.227668 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.227638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577"
Jul 24 17:41:12.227786 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.227673    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:41:12.241484 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.241453    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:12.304755 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.304391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:12.624313 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.624282    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:12.712603 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.712556    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:12.831681 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:12.831646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:13.026918 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.026847    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:13.285845 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.285812    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f77f7865d-vjglq"]
Jul 24 17:41:13.289898 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:41:13.289865    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd108987_7a83_4e8e_bd7a_3aecc3385d87.slice/crio-1ae50bc0f158d703130b771cee2551d6af0e4258dc3eec374cdb5602c157cbb2 WatchSource:0}: Error finding container 1ae50bc0f158d703130b771cee2551d6af0e4258dc3eec374cdb5602c157cbb2: Status 404 returned error can't find the container with id 1ae50bc0f158d703130b771cee2551d6af0e4258dc3eec374cdb5602c157cbb2
Jul 24 17:41:13.300422 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.300367    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-v8577"]
Jul 24 17:41:13.303631 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:41:13.303542    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd497c07a_1396_47b5_9c37_0d91fc12c7cf.slice/crio-7afaf10922b70564451a25f5ac446601a325952188e8626e1cd8246d70ee43dd WatchSource:0}: Error finding container 7afaf10922b70564451a25f5ac446601a325952188e8626e1cd8246d70ee43dd: Status 404 returned error can't find the container with id 7afaf10922b70564451a25f5ac446601a325952188e8626e1cd8246d70ee43dd
Jul 24 17:41:13.321840 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.321739    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cxc52"]
Jul 24 17:41:13.324941 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:41:13.324921    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01447c5b_a7fc_43e5_b0f0_a0c43ee232d9.slice/crio-012962deec1145615e0ab9e2d7adb9da7e081c95636ae148d89a3466a8f5aa27 WatchSource:0}: Error finding container 012962deec1145615e0ab9e2d7adb9da7e081c95636ae148d89a3466a8f5aa27: Status 404 returned error can't find the container with id 012962deec1145615e0ab9e2d7adb9da7e081c95636ae148d89a3466a8f5aa27
Jul 24 17:41:13.372066 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.372020    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:13.588213 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.588152    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9" podUID="76ced486-f7d7-4c74-8280-852c37e6b2ae" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/readyz\": dial tcp 10.132.0.9:8000: connect: connection refused"
Jul 24 17:41:13.936615 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.936513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577" event={"ID":"d497c07a-1396-47b5-9c37-0d91fc12c7cf","Type":"ContainerStarted","Data":"7afaf10922b70564451a25f5ac446601a325952188e8626e1cd8246d70ee43dd"}
Jul 24 17:41:13.937969 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.937938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cxc52" event={"ID":"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9","Type":"ContainerStarted","Data":"012962deec1145615e0ab9e2d7adb9da7e081c95636ae148d89a3466a8f5aa27"}
Jul 24 17:41:13.939741 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.939713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f77f7865d-vjglq" event={"ID":"cd108987-7a83-4e8e-bd7a-3aecc3385d87","Type":"ContainerStarted","Data":"401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889"}
Jul 24 17:41:13.939874 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.939758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f77f7865d-vjglq" event={"ID":"cd108987-7a83-4e8e-bd7a-3aecc3385d87","Type":"ContainerStarted","Data":"1ae50bc0f158d703130b771cee2551d6af0e4258dc3eec374cdb5602c157cbb2"}
Jul 24 17:41:13.939874 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.939810    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:41:13.942236 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.942148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fchlk" event={"ID":"d812cd21-578e-4bb6-884c-c452e3b2c174","Type":"ContainerStarted","Data":"5951e13f4a0dc0ce06fb202536d695286befc4168e624049e642ddc8b6f1e99f"}
Jul 24 17:41:13.948698 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.946825    2581 generic.go:356] "Generic (PLEG): container finished" podID="76ced486-f7d7-4c74-8280-852c37e6b2ae" containerID="d596d84fc99b50f2a8965d9533591a949195d9250c9b8957303a0ee088ead44b" exitCode=1
Jul 24 17:41:13.948698 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.946921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9" event={"ID":"76ced486-f7d7-4c74-8280-852c37e6b2ae","Type":"ContainerDied","Data":"d596d84fc99b50f2a8965d9533591a949195d9250c9b8957303a0ee088ead44b"}
Jul 24 17:41:13.948698 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.947003    2581 scope.go:117] "RemoveContainer" containerID="001f28737f1d6f1ee064b49da4e50c1006fc2a62fc5a22d6a12bb61c55a0d9b0"
Jul 24 17:41:13.948698 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.947402    2581 scope.go:117] "RemoveContainer" containerID="d596d84fc99b50f2a8965d9533591a949195d9250c9b8957303a0ee088ead44b"
Jul 24 17:41:13.948698 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:41:13.947677    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-658dff84cb-wz4l9_open-cluster-management-agent-addon(76ced486-f7d7-4c74-8280-852c37e6b2ae)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9" podUID="76ced486-f7d7-4c74-8280-852c37e6b2ae"
Jul 24 17:41:13.951926 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.951325    2581 generic.go:356] "Generic (PLEG): container finished" podID="c1e41958-e34b-45d6-bce4-d6026af4380e" containerID="45b240faecc72686898203896f8589c835388e33b6fe655c38b04627eac79951" exitCode=255
Jul 24 17:41:13.951926 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.951365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv" event={"ID":"c1e41958-e34b-45d6-bce4-d6026af4380e","Type":"ContainerDied","Data":"45b240faecc72686898203896f8589c835388e33b6fe655c38b04627eac79951"}
Jul 24 17:41:13.951926 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.951733    2581 scope.go:117] "RemoveContainer" containerID="45b240faecc72686898203896f8589c835388e33b6fe655c38b04627eac79951"
Jul 24 17:41:13.951926 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:41:13.951871    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-586c747fc8-wvmsv_open-cluster-management-agent-addon(c1e41958-e34b-45d6-bce4-d6026af4380e)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv" podUID="c1e41958-e34b-45d6-bce4-d6026af4380e"
Jul 24 17:41:13.963854 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:13.963808    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f77f7865d-vjglq" podStartSLOduration=286.963795859 podStartE2EDuration="4m46.963795859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:13.96196202 +0000 UTC m=+287.478473055" watchObservedRunningTime="2026-07-24 17:41:13.963795859 +0000 UTC m=+287.480306891"
Jul 24 17:41:14.330015 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:14.329923    2581 scope.go:117] "RemoveContainer" containerID="1306f5482a32d1a452eafddd5533211959dae3a62d326fba200dab565e33dbca"
Jul 24 17:41:14.638479 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:14.638386    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:14.960977 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:14.960800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577" event={"ID":"d497c07a-1396-47b5-9c37-0d91fc12c7cf","Type":"ContainerStarted","Data":"9a53109f8522953bf36fad2a5b6ec9c6213659dbdda2cfcb855912eda1e04cbd"}
Jul 24 17:41:14.976548 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:14.976440    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v8577" podStartSLOduration=264.909372492 podStartE2EDuration="4m25.976422828s" totalImagesPullingTime="1.067050336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:14.974824349 +0000 UTC m=+288.491335387" watchObservedRunningTime="2026-07-24 17:41:14.976422828 +0000 UTC m=+288.492933864"
Jul 24 17:41:15.023442 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:15.023340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:15.124153 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:15.124111    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:15.176551 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:15.176515    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:16.383650 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:16.383611    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:16.470644 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:16.470613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:16.654602 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:16.654507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:16.970114 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:16.970088    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cxc52" event={"ID":"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9","Type":"ContainerStarted","Data":"0c656a1894e5f32cf1230750810ab35319d57589a65f5b317e951316cfea43d4"}
Jul 24 17:41:16.972135 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:16.972111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fchlk" event={"ID":"d812cd21-578e-4bb6-884c-c452e3b2c174","Type":"ContainerStarted","Data":"c54e405133e03bc8cf0fd9744eb98318a8e91664d2074a24b56c58ce15feaefe"}
Jul 24 17:41:16.991642 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:16.991593    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fchlk" podStartSLOduration=17.139906493 podStartE2EDuration="28.991550367s" totalImagesPullingTime="11.851643874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:40:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:16.990390044 +0000 UTC m=+290.506901095" watchObservedRunningTime="2026-07-24 17:41:16.991550367 +0000 UTC m=+290.508061404"
Jul 24 17:41:17.976558 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:17.976513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cxc52" event={"ID":"01447c5b-a7fc-43e5-b0f0-a0c43ee232d9","Type":"ContainerStarted","Data":"bdf11d7c38ec1e667f30486dca4f49d8e0826287fe32a6654b3e995ede8d89d7"}
Jul 24 17:41:17.976972 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:17.976641    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cxc52"
Jul 24 17:41:17.994028 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:17.993982    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cxc52" podStartSLOduration=253.4373831 podStartE2EDuration="4m16.993969586s" totalImagesPullingTime="3.556586486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:17.992589978 +0000 UTC m=+291.509101007" watchObservedRunningTime="2026-07-24 17:41:17.993969586 +0000 UTC m=+291.510480621"
Jul 24 17:41:18.596391 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:18.596361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:20.538737 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.538702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:20.756384 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.756353    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:20.864600 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.864558    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fbr8m"]
Jul 24 17:41:20.867744 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.867730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:20.869996 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.869976    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 17:41:20.870370 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.870353    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 17:41:20.870450 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.870436    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 17:41:20.870510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.870451    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 17:41:20.870510 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.870460    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xfx6g\""
Jul 24 17:41:20.871129 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.871114    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 17:41:20.871129 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.871129    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 17:41:20.877492 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.877474    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:20.971506 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:20.971480    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:21.018404 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.018372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.018520 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.018405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nnd8\" (UniqueName: \"kubernetes.io/projected/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-kube-api-access-9nnd8\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.018520 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.018432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.018629 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.018512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-tls\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.018629 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.018541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-metrics-client-ca\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.018629 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.018561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-sys\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.018629 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.018600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-textfile\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.018629 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.018617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-root\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.018795 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.018632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-wtmp\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.095676 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.095639    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:21.119787 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.119727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.119787 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.119757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnd8\" (UniqueName: \"kubernetes.io/projected/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-kube-api-access-9nnd8\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.119931 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.119784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.119931 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.119815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-tls\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.119931 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:41:21.119900    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 17:41:21.120078 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.119939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-metrics-client-ca\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.120078 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:41:21.119959    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-tls podName:7cba1bb4-6b1e-4e24-8738-1fafbb602fef nodeName:}" failed. No retries permitted until 2026-07-24 17:41:21.619938103 +0000 UTC m=+295.136449118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-tls") pod "node-exporter-fbr8m" (UID: "7cba1bb4-6b1e-4e24-8738-1fafbb602fef") : secret "node-exporter-tls" not found
Jul 24 17:41:21.120078 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.119988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-sys\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.120078 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.120019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-textfile\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.120078 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.120042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-root\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.120078 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.120068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-wtmp\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.120346 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.120087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-sys\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.120346 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.120204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-root\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.120346 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.120298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-wtmp\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.120451 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.120430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-textfile\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.120514 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.120498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.120545 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.120531    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-metrics-client-ca\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.122060 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.122041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.128694 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.128674    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnd8\" (UniqueName: \"kubernetes.io/projected/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-kube-api-access-9nnd8\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.271018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.270982    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:21.623853 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.623818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-tls\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:21.624221 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:41:21.623978    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 17:41:21.624221 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:41:21.624046    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-tls podName:7cba1bb4-6b1e-4e24-8738-1fafbb602fef nodeName:}" failed. No retries permitted until 2026-07-24 17:41:22.624030535 +0000 UTC m=+296.140541553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-tls") pod "node-exporter-fbr8m" (UID: "7cba1bb4-6b1e-4e24-8738-1fafbb602fef") : secret "node-exporter-tls" not found
Jul 24 17:41:21.776431 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.776401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:21.873823 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.873793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:21.951207 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:21.951134    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:22.259686 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:22.259612    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv"
Jul 24 17:41:22.259944 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:22.259930    2581 scope.go:117] "RemoveContainer" containerID="45b240faecc72686898203896f8589c835388e33b6fe655c38b04627eac79951"
Jul 24 17:41:22.260087 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:41:22.260068    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-586c747fc8-wvmsv_open-cluster-management-agent-addon(c1e41958-e34b-45d6-bce4-d6026af4380e)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv" podUID="c1e41958-e34b-45d6-bce4-d6026af4380e"
Jul 24 17:41:22.272831 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:22.272806    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:41:22.273113 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:22.273101    2581 scope.go:117] "RemoveContainer" containerID="d596d84fc99b50f2a8965d9533591a949195d9250c9b8957303a0ee088ead44b"
Jul 24 17:41:22.273251 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:41:22.273231    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-658dff84cb-wz4l9_open-cluster-management-agent-addon(76ced486-f7d7-4c74-8280-852c37e6b2ae)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9" podUID="76ced486-f7d7-4c74-8280-852c37e6b2ae"
Jul 24 17:41:22.353789 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:22.353754    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:22.522483 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:22.522390    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:22.631690 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:22.631660    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-tls\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:22.633931 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:22.633907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cba1bb4-6b1e-4e24-8738-1fafbb602fef-node-exporter-tls\") pod \"node-exporter-fbr8m\" (UID: \"7cba1bb4-6b1e-4e24-8738-1fafbb602fef\") " pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:22.677940 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:22.677909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fbr8m"
Jul 24 17:41:22.692073 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:41:22.692047    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cba1bb4_6b1e_4e24_8738_1fafbb602fef.slice/crio-3f2200256588a100c6b4c218699c636132e4387446e3bdbc185de8520d82e4bc WatchSource:0}: Error finding container 3f2200256588a100c6b4c218699c636132e4387446e3bdbc185de8520d82e4bc: Status 404 returned error can't find the container with id 3f2200256588a100c6b4c218699c636132e4387446e3bdbc185de8520d82e4bc
Jul 24 17:41:22.992150 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:22.992121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbr8m" event={"ID":"7cba1bb4-6b1e-4e24-8738-1fafbb602fef","Type":"ContainerStarted","Data":"3f2200256588a100c6b4c218699c636132e4387446e3bdbc185de8520d82e4bc"}
Jul 24 17:41:23.587496 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:23.587473    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:41:23.587921 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:23.587906    2581 scope.go:117] "RemoveContainer" containerID="d596d84fc99b50f2a8965d9533591a949195d9250c9b8957303a0ee088ead44b"
Jul 24 17:41:23.723105 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:23.723081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:23.825867 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:23.825805    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:23.956985 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:41:23.956948    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6fck2" podUID="e2551d48-cd18-4944-b555-c595868e3881"
Jul 24 17:41:23.998154 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:23.998124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9" event={"ID":"76ced486-f7d7-4c74-8280-852c37e6b2ae","Type":"ContainerStarted","Data":"6cfe2f72eb4602318aa7d21e283019dad56db8925ecb7b72e4c5746ec7b5a857"}
Jul 24 17:41:23.998426 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:23.998406    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:41:23.999507 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:23.999482    2581 generic.go:356] "Generic (PLEG): container finished" podID="7cba1bb4-6b1e-4e24-8738-1fafbb602fef" containerID="87c44b086110553b61b2ef8ed88a9975cfe3843baf5948a38856b213a6901648" exitCode=0
Jul 24 17:41:23.999640 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:23.999528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbr8m" event={"ID":"7cba1bb4-6b1e-4e24-8738-1fafbb602fef","Type":"ContainerDied","Data":"87c44b086110553b61b2ef8ed88a9975cfe3843baf5948a38856b213a6901648"}
Jul 24 17:41:23.999963 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:23.999944    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658dff84cb-wz4l9"
Jul 24 17:41:24.120246 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:24.120223    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:24.221787 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:24.221756    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:25.004734 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:25.004699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbr8m" event={"ID":"7cba1bb4-6b1e-4e24-8738-1fafbb602fef","Type":"ContainerStarted","Data":"3577af2fde24db03d49fdf42a14914c1c9c4e57309b02d1a5986f0a094137552"}
Jul 24 17:41:25.005191 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:25.004751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbr8m" event={"ID":"7cba1bb4-6b1e-4e24-8738-1fafbb602fef","Type":"ContainerStarted","Data":"0d728f44a5dec1db696f7f57b096b6aefd86f658c04a31bfd10f78c1587f30ba"}
Jul 24 17:41:25.023701 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:25.023653    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fbr8m" podStartSLOduration=4.079921567 podStartE2EDuration="5.023637738s" totalImagesPullingTime="877.1347ms" totalInitContainerRuntime="66.581471ms" isStatefulPod=true podCreationTimestamp="2026-07-24 17:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:25.022082822 +0000 UTC m=+298.538593858" watchObservedRunningTime="2026-07-24 17:41:25.023637738 +0000 UTC m=+298.540148774"
Jul 24 17:41:25.552943 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:25.552913    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:25.612421 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:25.612391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:25.869148 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:25.869113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:26.639780 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:26.639751    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:26.739905 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:26.739877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:26.813959 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:26.813934    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 17:41:27.021842 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:27.021677    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:27.981420 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:27.981393    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cxc52"
Jul 24 17:41:30.647001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:30.646966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:31.692259 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:31.692227    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:32.231727 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:32.231697    2581 patch_prober.go:28] interesting pod/image-registry-f77f7865d-vjglq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 24 17:41:32.231876 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:32.231741    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-f77f7865d-vjglq" podUID="cd108987-7a83-4e8e-bd7a-3aecc3385d87" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 17:41:32.952716 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:32.952687    2581 scope.go:117] "RemoveContainer" containerID="45b240faecc72686898203896f8589c835388e33b6fe655c38b04627eac79951"
Jul 24 17:41:34.033706 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:34.033671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586c747fc8-wvmsv" event={"ID":"c1e41958-e34b-45d6-bce4-d6026af4380e","Type":"ContainerStarted","Data":"756ee069f0ebb8b071f9c12ed5fbb18234d340a350e93656030d18564e98ef28"}
Jul 24 17:41:34.389211 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:34.389180    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f77f7865d-vjglq"]
Jul 24 17:41:34.393858 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:34.393833    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:41:34.464447 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:34.464417    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:34.565008 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:34.564979    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:34.665531 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:34.665459    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:34.713176 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:34.713121    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:35.952474 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:35.952448    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:41:36.565117 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:36.565086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:36.618693 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:36.618667    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:38.686681 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:38.686649    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:38.780785 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:38.780756    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:39.849081 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:39.849043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:39.950524 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:39.950489    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:41.624022 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:41.623985    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:41.676785 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:41.676743    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:41.759083 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:41.759049    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:45.024332 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:45.024301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:45.124184 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:45.124156    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:46.237585 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:46.237535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:46.291807 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:46.291781    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:51.340695 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:51.340664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:51.442370 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:51.442337    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:55.316009 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:55.315974    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:55.372215 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:55.372176    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:59.407715 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.407653    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f77f7865d-vjglq" podUID="cd108987-7a83-4e8e-bd7a-3aecc3385d87" containerName="registry" containerID="cri-o://401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889" gracePeriod=30
Jul 24 17:41:59.641409 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.641387    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:41:59.721856 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.721788    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd108987-7a83-4e8e-bd7a-3aecc3385d87-image-registry-private-configuration\") pod \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") "
Jul 24 17:41:59.721856 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.721830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd108987-7a83-4e8e-bd7a-3aecc3385d87-ca-trust-extracted\") pod \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") "
Jul 24 17:41:59.722033 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.721859    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd108987-7a83-4e8e-bd7a-3aecc3385d87-installation-pull-secrets\") pod \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") "
Jul 24 17:41:59.722033 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.721893    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw69c\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-kube-api-access-jw69c\") pod \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") "
Jul 24 17:41:59.722033 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.721919    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd108987-7a83-4e8e-bd7a-3aecc3385d87-trusted-ca\") pod \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") "
Jul 24 17:41:59.722033 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.721946    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-certificates\") pod \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") "
Jul 24 17:41:59.722033 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.721995    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") pod \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") "
Jul 24 17:41:59.722033 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.722023    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-bound-sa-token\") pod \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\" (UID: \"cd108987-7a83-4e8e-bd7a-3aecc3385d87\") "
Jul 24 17:41:59.722461 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.722425    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cd108987-7a83-4e8e-bd7a-3aecc3385d87" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:41:59.722652 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.722611    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd108987-7a83-4e8e-bd7a-3aecc3385d87-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cd108987-7a83-4e8e-bd7a-3aecc3385d87" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:41:59.724772 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.724738    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-kube-api-access-jw69c" (OuterVolumeSpecName: "kube-api-access-jw69c") pod "cd108987-7a83-4e8e-bd7a-3aecc3385d87" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87"). InnerVolumeSpecName "kube-api-access-jw69c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:41:59.724876 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.724750    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd108987-7a83-4e8e-bd7a-3aecc3385d87-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cd108987-7a83-4e8e-bd7a-3aecc3385d87" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:41:59.724876 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.724795    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cd108987-7a83-4e8e-bd7a-3aecc3385d87" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:41:59.724876 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.724807    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cd108987-7a83-4e8e-bd7a-3aecc3385d87" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:41:59.724876 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.724844    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd108987-7a83-4e8e-bd7a-3aecc3385d87-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cd108987-7a83-4e8e-bd7a-3aecc3385d87" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:41:59.730541 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.730515    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd108987-7a83-4e8e-bd7a-3aecc3385d87-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cd108987-7a83-4e8e-bd7a-3aecc3385d87" (UID: "cd108987-7a83-4e8e-bd7a-3aecc3385d87"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:41:59.823155 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.823126    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-certificates\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 17:41:59.823155 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.823150    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-registry-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 17:41:59.823155 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.823159    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-bound-sa-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 17:41:59.823360 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.823167    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd108987-7a83-4e8e-bd7a-3aecc3385d87-image-registry-private-configuration\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 17:41:59.823360 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.823177    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd108987-7a83-4e8e-bd7a-3aecc3385d87-ca-trust-extracted\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 17:41:59.823360 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.823185    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd108987-7a83-4e8e-bd7a-3aecc3385d87-installation-pull-secrets\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 17:41:59.823360 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.823194    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw69c\" (UniqueName: \"kubernetes.io/projected/cd108987-7a83-4e8e-bd7a-3aecc3385d87-kube-api-access-jw69c\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 17:41:59.823360 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:41:59.823211    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd108987-7a83-4e8e-bd7a-3aecc3385d87-trusted-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 17:42:00.042095 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.042021    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:00.122362 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.122330    2581 generic.go:356] "Generic (PLEG): container finished" podID="cd108987-7a83-4e8e-bd7a-3aecc3385d87" containerID="401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889" exitCode=0
Jul 24 17:42:00.122502 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.122371    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f77f7865d-vjglq"
Jul 24 17:42:00.122502 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.122415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f77f7865d-vjglq" event={"ID":"cd108987-7a83-4e8e-bd7a-3aecc3385d87","Type":"ContainerDied","Data":"401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889"}
Jul 24 17:42:00.122502 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.122467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f77f7865d-vjglq" event={"ID":"cd108987-7a83-4e8e-bd7a-3aecc3385d87","Type":"ContainerDied","Data":"1ae50bc0f158d703130b771cee2551d6af0e4258dc3eec374cdb5602c157cbb2"}
Jul 24 17:42:00.122502 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.122488    2581 scope.go:117] "RemoveContainer" containerID="401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889"
Jul 24 17:42:00.130982 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.130954    2581 scope.go:117] "RemoveContainer" containerID="401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889"
Jul 24 17:42:00.131213 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:42:00.131195    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889\": container with ID starting with 401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889 not found: ID does not exist" containerID="401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889"
Jul 24 17:42:00.131260 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.131220    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889"} err="failed to get container status \"401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889\": rpc error: code = NotFound desc = could not find container \"401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889\": container with ID starting with 401a92ec5eeb9d506db6028db73de3dbff91607ed734f82ca6d99f9d7e0a4889 not found: ID does not exist"
Jul 24 17:42:00.141408 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.141382    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:00.146729 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.146713    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f77f7865d-vjglq"]
Jul 24 17:42:00.153550 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.153531    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f77f7865d-vjglq"]
Jul 24 17:42:00.368115 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.368087    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:00.467798 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.467771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:00.956114 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:00.956085    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd108987-7a83-4e8e-bd7a-3aecc3385d87" path="/var/lib/kubelet/pods/cd108987-7a83-4e8e-bd7a-3aecc3385d87/volumes"
Jul 24 17:42:02.294964 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:02.294920    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t" podUID="f39df8d3-ff04-4e26-afe5-a3583037cf0f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 24 17:42:12.294773 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:12.294735    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t" podUID="f39df8d3-ff04-4e26-afe5-a3583037cf0f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 24 17:42:15.030480 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:15.030452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:15.086216 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:15.086183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:19.078082 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:19.077999    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:22.294696 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:22.294655    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t" podUID="f39df8d3-ff04-4e26-afe5-a3583037cf0f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 24 17:42:22.295050 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:22.294732    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t"
Jul 24 17:42:22.295159 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:22.295141    2581 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6f1849e00f43584ec7fb6891e95802b16046e7cf572bd0518117955994aef8f0"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 24 17:42:22.295195 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:22.295177    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t" podUID="f39df8d3-ff04-4e26-afe5-a3583037cf0f" containerName="service-proxy" containerID="cri-o://6f1849e00f43584ec7fb6891e95802b16046e7cf572bd0518117955994aef8f0" gracePeriod=30
Jul 24 17:42:23.202006 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:23.201974    2581 generic.go:356] "Generic (PLEG): container finished" podID="f39df8d3-ff04-4e26-afe5-a3583037cf0f" containerID="6f1849e00f43584ec7fb6891e95802b16046e7cf572bd0518117955994aef8f0" exitCode=2
Jul 24 17:42:23.202171 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:23.202021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t" event={"ID":"f39df8d3-ff04-4e26-afe5-a3583037cf0f","Type":"ContainerDied","Data":"6f1849e00f43584ec7fb6891e95802b16046e7cf572bd0518117955994aef8f0"}
Jul 24 17:42:23.202171 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:23.202062    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbffccb9b-sw79t" event={"ID":"f39df8d3-ff04-4e26-afe5-a3583037cf0f","Type":"ContainerStarted","Data":"7bfc12572396b1299b53889a50ce32e5dde8e6684535f6c6000e694bf71ec81e"}
Jul 24 17:42:25.297659 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:25.297624    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:25.394553 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:25.394522    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:40.048260 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:40.048228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:40.571762 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:40.571731    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:40.919588 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:40.919489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:42:40.921735 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:40.921718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2551d48-cd18-4944-b555-c595868e3881-metrics-certs\") pod \"network-metrics-daemon-6fck2\" (UID: \"e2551d48-cd18-4944-b555-c595868e3881\") " pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:42:41.055388 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:41.055357    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lvv7h\""
Jul 24 17:42:41.063286 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:41.063270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fck2"
Jul 24 17:42:41.230331 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:41.230253    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6fck2"]
Jul 24 17:42:41.234402 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:42:41.234378    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2551d48_cd18_4944_b555_c595868e3881.slice/crio-8340425e244bbb057c36a6f48dc303d9369f7b02c7c5ba4fc1b5da7d761081dd WatchSource:0}: Error finding container 8340425e244bbb057c36a6f48dc303d9369f7b02c7c5ba4fc1b5da7d761081dd: Status 404 returned error can't find the container with id 8340425e244bbb057c36a6f48dc303d9369f7b02c7c5ba4fc1b5da7d761081dd
Jul 24 17:42:41.236153 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:41.236138    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:42:41.258354 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:41.258327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6fck2" event={"ID":"e2551d48-cd18-4944-b555-c595868e3881","Type":"ContainerStarted","Data":"8340425e244bbb057c36a6f48dc303d9369f7b02c7c5ba4fc1b5da7d761081dd"}
Jul 24 17:42:43.266722 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:43.266684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6fck2" event={"ID":"e2551d48-cd18-4944-b555-c595868e3881","Type":"ContainerStarted","Data":"9df489d5e1fe9faa979b2f66736cd2b5cf88fc3742795630bf0fb21bab2a3c5c"}
Jul 24 17:42:43.267077 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:43.266737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6fck2" event={"ID":"e2551d48-cd18-4944-b555-c595868e3881","Type":"ContainerStarted","Data":"4f020ad3a266a2676d403f7d88a023c69f68df83545a53d54c1cc2e1a05412d2"}
Jul 24 17:42:43.283936 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:43.283889    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6fck2" podStartSLOduration=375.175506756 podStartE2EDuration="6m16.283874492s" totalImagesPullingTime="1.108367736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:43.283249706 +0000 UTC m=+376.799760745" watchObservedRunningTime="2026-07-24 17:42:43.283874492 +0000 UTC m=+376.800385528"
Jul 24 17:42:43.373771 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:43.373742    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:43.434621 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:43.434594    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:48.595182 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:48.595146    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:48.717621 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:48.717565    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:48.805136 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:48.805105    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:49.310558 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:49.310527    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:49.366264 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:49.366234    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:56.893431 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:56.893379    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:56.999218 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:56.999186    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:57.710798 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:57.710767    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:57.780195 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:57.780171    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:59.967262 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:42:59.967232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:00.068626 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:43:00.068565    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:00.169825 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:43:00.169798    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:55.290158 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:43:55.290084    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:55.391670 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:43:55.391639    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:55.459689 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:43:55.459656    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:00.283205 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:00.283172    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:00.337482 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:00.337452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:00.421766 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:00.421741    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:00.478006 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:00.477974    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:01.030904 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:01.030874    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:07.327546 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:07.327513    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:07.381983 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:07.381951    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:10.082631 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:10.082594    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:10.184024 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:10.183998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:11.220322 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:11.220284    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:11.276648 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:11.276624    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:11.412308 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:11.412279    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:11.471496 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:11.471432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:24.639465 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:24.639433    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:48.706609 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:44:48.706544    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:09.096002 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:09.095966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:09.192832 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:09.192793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:31.341092 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:31.341048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:32.633607 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:32.633557    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:35.399554 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:35.399515    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:35.501528 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:35.501488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:35.872587 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:35.872544    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:36.219654 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:36.219535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:37.039446 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:37.039411    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:37.132829 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:37.132794    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:39.282999 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:39.282970    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:39.435545 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:39.435515    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:42.342385 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:42.342354    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:42.443100 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:42.443070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:44.631250 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:44.631217    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:44.702520 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:44.702496    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:45.368558 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.368521    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8"]
Jul 24 17:45:45.368850 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.368833    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd108987-7a83-4e8e-bd7a-3aecc3385d87" containerName="registry"
Jul 24 17:45:45.368928 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.368853    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd108987-7a83-4e8e-bd7a-3aecc3385d87" containerName="registry"
Jul 24 17:45:45.368980 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.368933    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd108987-7a83-4e8e-bd7a-3aecc3385d87" containerName="registry"
Jul 24 17:45:45.373285 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.373248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8"
Jul 24 17:45:45.376492 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.376463    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 24 17:45:45.376615 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.376462    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zjkb5\""
Jul 24 17:45:45.376615 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.376493    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 24 17:45:45.382764 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.382742    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8"]
Jul 24 17:45:45.434975 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.434950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4ac3b23-b8e9-4044-b724-3cad2a559b71-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6m6c8\" (UID: \"b4ac3b23-b8e9-4044-b724-3cad2a559b71\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8"
Jul 24 17:45:45.435108 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.435006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjbbv\" (UniqueName: \"kubernetes.io/projected/b4ac3b23-b8e9-4044-b724-3cad2a559b71-kube-api-access-zjbbv\") pod \"cert-manager-cainjector-559cdd8c87-6m6c8\" (UID: \"b4ac3b23-b8e9-4044-b724-3cad2a559b71\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8"
Jul 24 17:45:45.536134 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.536096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4ac3b23-b8e9-4044-b724-3cad2a559b71-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6m6c8\" (UID: \"b4ac3b23-b8e9-4044-b724-3cad2a559b71\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8"
Jul 24 17:45:45.536309 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.536161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjbbv\" (UniqueName: \"kubernetes.io/projected/b4ac3b23-b8e9-4044-b724-3cad2a559b71-kube-api-access-zjbbv\") pod \"cert-manager-cainjector-559cdd8c87-6m6c8\" (UID: \"b4ac3b23-b8e9-4044-b724-3cad2a559b71\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8"
Jul 24 17:45:45.545027 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.544993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4ac3b23-b8e9-4044-b724-3cad2a559b71-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6m6c8\" (UID: \"b4ac3b23-b8e9-4044-b724-3cad2a559b71\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8"
Jul 24 17:45:45.545214 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.545193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjbbv\" (UniqueName: \"kubernetes.io/projected/b4ac3b23-b8e9-4044-b724-3cad2a559b71-kube-api-access-zjbbv\") pod \"cert-manager-cainjector-559cdd8c87-6m6c8\" (UID: \"b4ac3b23-b8e9-4044-b724-3cad2a559b71\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8"
Jul 24 17:45:45.681828 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.681759    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8"
Jul 24 17:45:45.811221 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.811189    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8"]
Jul 24 17:45:45.814961 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:45:45.814931    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4ac3b23_b8e9_4044_b724_3cad2a559b71.slice/crio-b2185e2ed84af77e5437194c19b9ecacf220c2cb110aca3c8d3af5594d41c253 WatchSource:0}: Error finding container b2185e2ed84af77e5437194c19b9ecacf220c2cb110aca3c8d3af5594d41c253: Status 404 returned error can't find the container with id b2185e2ed84af77e5437194c19b9ecacf220c2cb110aca3c8d3af5594d41c253
Jul 24 17:45:45.855758 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:45.855727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8" event={"ID":"b4ac3b23-b8e9-4044-b724-3cad2a559b71","Type":"ContainerStarted","Data":"b2185e2ed84af77e5437194c19b9ecacf220c2cb110aca3c8d3af5594d41c253"}
Jul 24 17:45:47.179149 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:47.179114    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:47.292929 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:47.292892    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:47.490053 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:47.489973    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:49.777130 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.777094    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j"]
Jul 24 17:45:49.780058 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.780041    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j"
Jul 24 17:45:49.782677 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.782657    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 17:45:49.782780 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.782657    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 17:45:49.783424 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.783408    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-dzptv\""
Jul 24 17:45:49.791935 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.791909    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j"]
Jul 24 17:45:49.871449 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.871409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64581efe-8049-44d3-baa4-e31fb0241b14-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qhs8j\" (UID: \"64581efe-8049-44d3-baa4-e31fb0241b14\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j"
Jul 24 17:45:49.871449 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.871448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8gk8\" (UniqueName: \"kubernetes.io/projected/64581efe-8049-44d3-baa4-e31fb0241b14-kube-api-access-l8gk8\") pod \"openshift-lws-operator-fd8ccff4c-qhs8j\" (UID: \"64581efe-8049-44d3-baa4-e31fb0241b14\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j"
Jul 24 17:45:49.872155 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.872127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8" event={"ID":"b4ac3b23-b8e9-4044-b724-3cad2a559b71","Type":"ContainerStarted","Data":"c0dc4e0ae03a059662cae3b0b70e65aa0108ab5bcea561c6d2031c453b4e22aa"}
Jul 24 17:45:49.892650 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.892607    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6m6c8" podStartSLOduration=1.493161515 podStartE2EDuration="4.892593431s" totalImagesPullingTime="3.399431916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:45:49.892528308 +0000 UTC m=+563.409039347" watchObservedRunningTime="2026-07-24 17:45:49.892593431 +0000 UTC m=+563.409104467"
Jul 24 17:45:49.972742 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.972714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64581efe-8049-44d3-baa4-e31fb0241b14-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qhs8j\" (UID: \"64581efe-8049-44d3-baa4-e31fb0241b14\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j"
Jul 24 17:45:49.972923 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.972748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8gk8\" (UniqueName: \"kubernetes.io/projected/64581efe-8049-44d3-baa4-e31fb0241b14-kube-api-access-l8gk8\") pod \"openshift-lws-operator-fd8ccff4c-qhs8j\" (UID: \"64581efe-8049-44d3-baa4-e31fb0241b14\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j"
Jul 24 17:45:49.973148 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.973127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64581efe-8049-44d3-baa4-e31fb0241b14-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qhs8j\" (UID: \"64581efe-8049-44d3-baa4-e31fb0241b14\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j"
Jul 24 17:45:49.980001 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.979980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8gk8\" (UniqueName: \"kubernetes.io/projected/64581efe-8049-44d3-baa4-e31fb0241b14-kube-api-access-l8gk8\") pod \"openshift-lws-operator-fd8ccff4c-qhs8j\" (UID: \"64581efe-8049-44d3-baa4-e31fb0241b14\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j"
Jul 24 17:45:49.994193 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:49.994167    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:50.087930 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:50.087904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j"
Jul 24 17:45:50.103205 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:50.103175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:50.214080 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:50.214058    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j"]
Jul 24 17:45:50.216649 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:45:50.216622    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64581efe_8049_44d3_baa4_e31fb0241b14.slice/crio-ee0945da4f1b9d3339a67271923437d476e857e3dfff738e13fe6ded1a2caa2d WatchSource:0}: Error finding container ee0945da4f1b9d3339a67271923437d476e857e3dfff738e13fe6ded1a2caa2d: Status 404 returned error can't find the container with id ee0945da4f1b9d3339a67271923437d476e857e3dfff738e13fe6ded1a2caa2d
Jul 24 17:45:50.876908 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:50.876869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j" event={"ID":"64581efe-8049-44d3-baa4-e31fb0241b14","Type":"ContainerStarted","Data":"ee0945da4f1b9d3339a67271923437d476e857e3dfff738e13fe6ded1a2caa2d"}
Jul 24 17:45:53.891375 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:53.891340    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j" event={"ID":"64581efe-8049-44d3-baa4-e31fb0241b14","Type":"ContainerStarted","Data":"9a3b028ffa65465f6c0f735447656a3a60c25c273345908afc83dce77ddd0a04"}
Jul 24 17:45:53.909677 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:53.909620    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhs8j" podStartSLOduration=2.037974841 podStartE2EDuration="4.909603123s" totalImagesPullingTime="2.871628282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:45:53.909090978 +0000 UTC m=+567.425602014" watchObservedRunningTime="2026-07-24 17:45:53.909603123 +0000 UTC m=+567.426114160"
Jul 24 17:45:54.652198 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:54.652146    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:54.788150 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:54.788119    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:56.775020 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:45:56.774982    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:08.337425 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:08.337390    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:08.487705 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:08.487671    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:08.958212 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:08.958182    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:09.129286 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.129257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:09.230077 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.229996    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:09.290656 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.290619    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"]
Jul 24 17:46:09.296702 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.296683    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.302174 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.302152    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 17:46:09.302374 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.302359    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 17:46:09.302431 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.302413    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 17:46:09.303929 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.303914    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 17:46:09.314400 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.314382    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jth4d\""
Jul 24 17:46:09.325548 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.325523    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"]
Jul 24 17:46:09.408224 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.408194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c3260159-f143-4de8-83fe-75c5fe49a976-manifests\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.408224 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.408226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65tww\" (UniqueName: \"kubernetes.io/projected/c3260159-f143-4de8-83fe-75c5fe49a976-kube-api-access-65tww\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.408676 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.408247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3260159-f143-4de8-83fe-75c5fe49a976-tmp\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.408676 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.408349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3260159-f143-4de8-83fe-75c5fe49a976-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.408676 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.408380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c3260159-f143-4de8-83fe-75c5fe49a976-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.485151 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.485059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:09.509167 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.509142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c3260159-f143-4de8-83fe-75c5fe49a976-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.509266 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.509180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c3260159-f143-4de8-83fe-75c5fe49a976-manifests\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.509266 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.509195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65tww\" (UniqueName: \"kubernetes.io/projected/c3260159-f143-4de8-83fe-75c5fe49a976-kube-api-access-65tww\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.509359 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.509334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3260159-f143-4de8-83fe-75c5fe49a976-tmp\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.509432 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.509401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3260159-f143-4de8-83fe-75c5fe49a976-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.509612 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.509591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c3260159-f143-4de8-83fe-75c5fe49a976-manifests\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.511540 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.511515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3260159-f143-4de8-83fe-75c5fe49a976-tmp\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.511801 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.511773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c3260159-f143-4de8-83fe-75c5fe49a976-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.511801 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.511795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3260159-f143-4de8-83fe-75c5fe49a976-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.521325 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.518652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65tww\" (UniqueName: \"kubernetes.io/projected/c3260159-f143-4de8-83fe-75c5fe49a976-kube-api-access-65tww\") pod \"opendatahub-operator-controller-manager-6b794f588d-qc7qd\" (UID: \"c3260159-f143-4de8-83fe-75c5fe49a976\") " pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.586674 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.586647    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:09.606720 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.606701    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:09.689463 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.689434    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:09.735792 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.735708    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"]
Jul 24 17:46:09.739619 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:46:09.739588    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3260159_f143_4de8_83fe_75c5fe49a976.slice/crio-885f93544810c869297da502883c43367fc930611bccfae547297604bf628612 WatchSource:0}: Error finding container 885f93544810c869297da502883c43367fc930611bccfae547297604bf628612: Status 404 returned error can't find the container with id 885f93544810c869297da502883c43367fc930611bccfae547297604bf628612
Jul 24 17:46:09.787114 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.787085    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:09.888167 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.888129    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:09.948565 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.948534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd" event={"ID":"c3260159-f143-4de8-83fe-75c5fe49a976","Type":"ContainerStarted","Data":"885f93544810c869297da502883c43367fc930611bccfae547297604bf628612"}
Jul 24 17:46:09.990121 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:09.990054    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:13.965554 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:13.965517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd" event={"ID":"c3260159-f143-4de8-83fe-75c5fe49a976","Type":"ContainerStarted","Data":"6275c17d1e24780ab9d4967f7866e43f6f21de608268707b1ebaea1d662b74ef"}
Jul 24 17:46:14.916206 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:14.916175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:14.969816 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:14.969785    2581 generic.go:356] "Generic (PLEG): container finished" podID="c3260159-f143-4de8-83fe-75c5fe49a976" containerID="6275c17d1e24780ab9d4967f7866e43f6f21de608268707b1ebaea1d662b74ef" exitCode=0
Jul 24 17:46:14.970242 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:14.969840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd" event={"ID":"c3260159-f143-4de8-83fe-75c5fe49a976","Type":"ContainerDied","Data":"6275c17d1e24780ab9d4967f7866e43f6f21de608268707b1ebaea1d662b74ef"}
Jul 24 17:46:15.035849 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:15.035817    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:15.974889 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:15.974852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd" event={"ID":"c3260159-f143-4de8-83fe-75c5fe49a976","Type":"ContainerStarted","Data":"c401ca43b3f52ebf937c696be26d5aa69a1d6a0fcac00812a5b06ae18398e900"}
Jul 24 17:46:15.975263 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:15.974993    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:16.002197 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:16.002150    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd" podStartSLOduration=2.353307243 podStartE2EDuration="7.002136047s" totalImagesPullingTime="3.572651973s" totalInitContainerRuntime="1.076176831s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:16.00081974 +0000 UTC m=+589.517330777" watchObservedRunningTime="2026-07-24 17:46:16.002136047 +0000 UTC m=+589.518647080"
Jul 24 17:46:24.558256 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:24.558219    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:26.979768 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:26.979740    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b794f588d-qc7qd"
Jul 24 17:46:28.767516 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:28.767480    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:28.879460 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:28.879428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:29.347671 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.347641    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:29.677923 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.677842    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"]
Jul 24 17:46:29.680966 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.680945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.684144 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.684127    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 17:46:29.684144 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.684139    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vtzzk\""
Jul 24 17:46:29.684396 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.684382    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 17:46:29.684614 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.684600    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 17:46:29.703417 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.703393    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"]
Jul 24 17:46:29.742155 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.742130    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:29.765446 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.765416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce937f90-f289-4673-8768-90dd0d57ac25-cert\") pod \"lws-controller-manager-5b4bb5cddb-m8qqp\" (UID: \"ce937f90-f289-4673-8768-90dd0d57ac25\") " pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.765563 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.765457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ce937f90-f289-4673-8768-90dd0d57ac25-manager-config\") pod \"lws-controller-manager-5b4bb5cddb-m8qqp\" (UID: \"ce937f90-f289-4673-8768-90dd0d57ac25\") " pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.765563 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.765480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvxbx\" (UniqueName: \"kubernetes.io/projected/ce937f90-f289-4673-8768-90dd0d57ac25-kube-api-access-vvxbx\") pod \"lws-controller-manager-5b4bb5cddb-m8qqp\" (UID: \"ce937f90-f289-4673-8768-90dd0d57ac25\") " pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.765698 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.765565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce937f90-f289-4673-8768-90dd0d57ac25-metrics-cert\") pod \"lws-controller-manager-5b4bb5cddb-m8qqp\" (UID: \"ce937f90-f289-4673-8768-90dd0d57ac25\") " pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.836533 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.836508    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:29.866895 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.866868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce937f90-f289-4673-8768-90dd0d57ac25-metrics-cert\") pod \"lws-controller-manager-5b4bb5cddb-m8qqp\" (UID: \"ce937f90-f289-4673-8768-90dd0d57ac25\") " pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.867018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.866921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce937f90-f289-4673-8768-90dd0d57ac25-cert\") pod \"lws-controller-manager-5b4bb5cddb-m8qqp\" (UID: \"ce937f90-f289-4673-8768-90dd0d57ac25\") " pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.867018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.866951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ce937f90-f289-4673-8768-90dd0d57ac25-manager-config\") pod \"lws-controller-manager-5b4bb5cddb-m8qqp\" (UID: \"ce937f90-f289-4673-8768-90dd0d57ac25\") " pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.867018 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.866972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxbx\" (UniqueName: \"kubernetes.io/projected/ce937f90-f289-4673-8768-90dd0d57ac25-kube-api-access-vvxbx\") pod \"lws-controller-manager-5b4bb5cddb-m8qqp\" (UID: \"ce937f90-f289-4673-8768-90dd0d57ac25\") " pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.867674 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.867652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ce937f90-f289-4673-8768-90dd0d57ac25-manager-config\") pod \"lws-controller-manager-5b4bb5cddb-m8qqp\" (UID: \"ce937f90-f289-4673-8768-90dd0d57ac25\") " pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.869240 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.869220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce937f90-f289-4673-8768-90dd0d57ac25-metrics-cert\") pod \"lws-controller-manager-5b4bb5cddb-m8qqp\" (UID: \"ce937f90-f289-4673-8768-90dd0d57ac25\") " pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.869415 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.869397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce937f90-f289-4673-8768-90dd0d57ac25-cert\") pod \"lws-controller-manager-5b4bb5cddb-m8qqp\" (UID: \"ce937f90-f289-4673-8768-90dd0d57ac25\") " pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.885565 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.885541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxbx\" (UniqueName: \"kubernetes.io/projected/ce937f90-f289-4673-8768-90dd0d57ac25-kube-api-access-vvxbx\") pod \"lws-controller-manager-5b4bb5cddb-m8qqp\" (UID: \"ce937f90-f289-4673-8768-90dd0d57ac25\") " pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:29.941278 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.941210    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:29.989852 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:29.989828    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:30.031368 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:30.031330    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:30.117959 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:30.117931    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"]
Jul 24 17:46:30.121156 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:46:30.121131    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce937f90_f289_4673_8768_90dd0d57ac25.slice/crio-8368d151ad8935c2d83fe19285a54da711df4dfc8355813d78510345e58583ca WatchSource:0}: Error finding container 8368d151ad8935c2d83fe19285a54da711df4dfc8355813d78510345e58583ca: Status 404 returned error can't find the container with id 8368d151ad8935c2d83fe19285a54da711df4dfc8355813d78510345e58583ca
Jul 24 17:46:30.668512 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:30.668479    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:30.844086 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:30.844051    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:30.946197 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:30.946120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:31.029604 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:31.029551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp" event={"ID":"ce937f90-f289-4673-8768-90dd0d57ac25","Type":"ContainerStarted","Data":"8368d151ad8935c2d83fe19285a54da711df4dfc8355813d78510345e58583ca"}
Jul 24 17:46:32.034404 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:32.034375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp" event={"ID":"ce937f90-f289-4673-8768-90dd0d57ac25","Type":"ContainerStarted","Data":"2bc4d805180884a364fe7cf11ec6154ce841477921618d8c18f0de674db5afe6"}
Jul 24 17:46:32.034735 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:32.034488    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:32.052796 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:32.052749    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp" podStartSLOduration=1.21171317 podStartE2EDuration="3.052733873s" totalImagesPullingTime="1.841020703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:32.05162966 +0000 UTC m=+605.568140696" watchObservedRunningTime="2026-07-24 17:46:32.052733873 +0000 UTC m=+605.569244909"
Jul 24 17:46:32.232400 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:32.232321    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:33.604649 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:33.604619    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:36.323877 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:36.323843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:36.425525 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:36.425495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:36.706344 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:36.706268    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:36.870479 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:36.870442    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:36.941700 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:36.941666    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:39.211979 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.211945    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"]
Jul 24 17:46:39.214871 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.214854    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.217469 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.217446    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 24 17:46:39.217771 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.217749    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 24 17:46:39.217771 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.217753    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 24 17:46:39.217975 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.217958    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-l5j77\""
Jul 24 17:46:39.218029 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.217995    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 17:46:39.218116 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.218098    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 17:46:39.218699 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.218684    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 24 17:46:39.229246 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.229227    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"]
Jul 24 17:46:39.342009 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.341979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/081313fb-605f-4b48-beaa-202a611482dc-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.342158 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.342017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/081313fb-605f-4b48-beaa-202a611482dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.342158 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.342050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb46t\" (UniqueName: \"kubernetes.io/projected/081313fb-605f-4b48-beaa-202a611482dc-kube-api-access-pb46t\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.342158 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.342087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081313fb-605f-4b48-beaa-202a611482dc-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.342158 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.342121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/081313fb-605f-4b48-beaa-202a611482dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.342158 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.342154    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/081313fb-605f-4b48-beaa-202a611482dc-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.342318 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.342177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/081313fb-605f-4b48-beaa-202a611482dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.443262 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.443231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/081313fb-605f-4b48-beaa-202a611482dc-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.443405 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.443273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/081313fb-605f-4b48-beaa-202a611482dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.443467 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.443409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb46t\" (UniqueName: \"kubernetes.io/projected/081313fb-605f-4b48-beaa-202a611482dc-kube-api-access-pb46t\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.443467 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.443458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081313fb-605f-4b48-beaa-202a611482dc-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.443599 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.443525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/081313fb-605f-4b48-beaa-202a611482dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.443663 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.443595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/081313fb-605f-4b48-beaa-202a611482dc-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.443663 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.443637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/081313fb-605f-4b48-beaa-202a611482dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.444089 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.444062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/081313fb-605f-4b48-beaa-202a611482dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.445788 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.445767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/081313fb-605f-4b48-beaa-202a611482dc-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.445888 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.445821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/081313fb-605f-4b48-beaa-202a611482dc-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.445934 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.445882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/081313fb-605f-4b48-beaa-202a611482dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.445934 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.445929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/081313fb-605f-4b48-beaa-202a611482dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.457887 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.457859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb46t\" (UniqueName: \"kubernetes.io/projected/081313fb-605f-4b48-beaa-202a611482dc-kube-api-access-pb46t\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.460964 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.460945    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081313fb-605f-4b48-beaa-202a611482dc-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-kzv5g\" (UID: \"081313fb-605f-4b48-beaa-202a611482dc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.524094 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.524041    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:39.682424 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:39.682399    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"]
Jul 24 17:46:39.683539 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:46:39.683512    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081313fb_605f_4b48_beaa_202a611482dc.slice/crio-46a66afd56d1f260cb297f443db762dac10077ae98aa99ef6ca3c2b628248430 WatchSource:0}: Error finding container 46a66afd56d1f260cb297f443db762dac10077ae98aa99ef6ca3c2b628248430: Status 404 returned error can't find the container with id 46a66afd56d1f260cb297f443db762dac10077ae98aa99ef6ca3c2b628248430
Jul 24 17:46:40.062486 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:40.062453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g" event={"ID":"081313fb-605f-4b48-beaa-202a611482dc","Type":"ContainerStarted","Data":"46a66afd56d1f260cb297f443db762dac10077ae98aa99ef6ca3c2b628248430"}
Jul 24 17:46:41.155526 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:41.155490    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:41.283082 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:41.283044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:41.416955 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:41.416875    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:41.489020 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:41.488971    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:42.156226 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:42.156184    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 17:46:42.156558 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:42.156262    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 17:46:42.651766 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:42.651726    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:42.747747 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:42.747711    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:43.040292 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:43.040210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b4bb5cddb-m8qqp"
Jul 24 17:46:43.077257 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:43.077224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g" event={"ID":"081313fb-605f-4b48-beaa-202a611482dc","Type":"ContainerStarted","Data":"f0aff7779190425f7a2c03fbfa6c01f18fdb853a0932b63e85709600f1138ee5"}
Jul 24 17:46:43.077445 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:43.077290    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:46:43.078772 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:43.078747    2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-kzv5g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 17:46:43.078889 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:43.078805    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g" podUID="081313fb-605f-4b48-beaa-202a611482dc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 17:46:43.153639 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:43.153558    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g" podStartSLOduration=1.6831201180000002 podStartE2EDuration="4.153534553s" totalImagesPullingTime="2.470414435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:43.151926961 +0000 UTC m=+616.668438000" watchObservedRunningTime="2026-07-24 17:46:43.153534553 +0000 UTC m=+616.670045589"
Jul 24 17:46:44.082719 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:44.082687    2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-kzv5g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 17:46:44.083151 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:44.082751    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g" podUID="081313fb-605f-4b48-beaa-202a611482dc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 17:46:47.083521 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:46:47.083497    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kzv5g"
Jul 24 17:47:00.907043 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:47:00.906969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:01.024125 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:47:01.024085    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:21.775684 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:47:21.775629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:21.871089 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:47:21.871059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:30.251740 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:47:30.251699    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:30.373543 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:47:30.373505    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:30.503646 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:47:30.503545    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:54.681924 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:47:54.681882    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:58.146015 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:47:58.145977    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:58.872148 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:47:58.872116    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:59.465687 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:47:59.465650    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:59.572280 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:47:59.572254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:01.577733 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:01.577699    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:02.111363 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:02.111334    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:03.986256 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:03.986222    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:05.699709 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:05.699675    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:07.274116 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:07.274079    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:08.513210 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:08.513177    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:10.478751 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:10.478709    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:10.585478 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:10.585446    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:10.693186 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:10.693157    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:10.793596 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:10.793494    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:13.460862 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:13.460827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:13.627237 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:13.627204    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:14.469164 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:14.469132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:14.567480 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:14.567451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:14.903249 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:14.903214    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:14.958143 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:14.958118    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:18.663655 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:18.663621    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:18.761709 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:18.761676    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:25.150129 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:25.150096    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:25.259866 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:25.259828    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:29.498458 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:29.498429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:29.602070 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:29.602039    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:33.297152 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:33.297112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:33.398888 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:33.398854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:33.496207 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:33.496178    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:33.558750 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:33.558722    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:38.302127 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:38.302087    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:38.372512 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:38.372483    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:43.977742 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:43.977705    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:44.078339 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:44.078311    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:54.888869 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:54.888834    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:55.094784 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:48:55.094745    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:09.389314 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:09.389276    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:09.520599 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:09.520552    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:09.623121 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:09.623090    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:10.289931 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:10.289902    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:10.393189 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:10.393157    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:10.956209 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:10.956180    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:11.083468 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:11.083439    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:28.246887 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:28.246850    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:28.352553 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:28.352511    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:28.795161 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:28.795123    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:41.272908 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:41.272865    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:41.415229 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:41.415197    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:49.131862 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:49.131776    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:49.226952 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:49.226919    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:49.327221 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:49.327187    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:52.177138 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:52.177109    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:52.278467 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:49:52.278427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:23.914047 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:23.914010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:24.003751 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:24.003723    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:24.526503 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:24.526467    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:25.017122 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:25.017093    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:25.115048 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:25.115018    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:25.362111 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:25.362081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:36.388519 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:36.388489    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:36.491231 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:36.491202    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:48.618669 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:48.618637    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:48.939878 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:48.939808    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"]
Jul 24 17:50:48.942913 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:48.942892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:48.945219 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:48.945192    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bs9cs\""
Jul 24 17:50:48.945342 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:48.945327    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 17:50:48.946080 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:48.946062    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 17:50:48.946181 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:48.946094    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 24 17:50:48.958831 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:48.958807    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"]
Jul 24 17:50:48.975857 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:48.975829    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:49.050021 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.049989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1b93aab-418a-45fe-9015-8200a69da666-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.050021 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.050020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1b93aab-418a-45fe-9015-8200a69da666-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.050213 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.050047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1b93aab-418a-45fe-9015-8200a69da666-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.050213 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.050130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1b93aab-418a-45fe-9015-8200a69da666-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.050213 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.050171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1b93aab-418a-45fe-9015-8200a69da666-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.050213 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.050191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjcd\" (UniqueName: \"kubernetes.io/projected/a1b93aab-418a-45fe-9015-8200a69da666-kube-api-access-9cjcd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.150782 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.150746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1b93aab-418a-45fe-9015-8200a69da666-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.150944 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.150802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1b93aab-418a-45fe-9015-8200a69da666-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.150944 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.150918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1b93aab-418a-45fe-9015-8200a69da666-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.151019 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.150964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjcd\" (UniqueName: \"kubernetes.io/projected/a1b93aab-418a-45fe-9015-8200a69da666-kube-api-access-9cjcd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.151053 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.151021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1b93aab-418a-45fe-9015-8200a69da666-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.151097 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.151048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1b93aab-418a-45fe-9015-8200a69da666-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.151205 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.151187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1b93aab-418a-45fe-9015-8200a69da666-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.151379 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.151350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1b93aab-418a-45fe-9015-8200a69da666-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.151379 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.151353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1b93aab-418a-45fe-9015-8200a69da666-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.153183 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.153164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1b93aab-418a-45fe-9015-8200a69da666-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.153345 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.153329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1b93aab-418a-45fe-9015-8200a69da666-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.159895 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.159872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjcd\" (UniqueName: \"kubernetes.io/projected/a1b93aab-418a-45fe-9015-8200a69da666-kube-api-access-9cjcd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht\" (UID: \"a1b93aab-418a-45fe-9015-8200a69da666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.253428 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.253357    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:49.380524 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.380498    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"]
Jul 24 17:50:49.382901 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:50:49.382874    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b93aab_418a_45fe_9015_8200a69da666.slice/crio-38819a34ca9a904cc34f79da9e956795288c136a2fe35db71272d2911fad764e WatchSource:0}: Error finding container 38819a34ca9a904cc34f79da9e956795288c136a2fe35db71272d2911fad764e: Status 404 returned error can't find the container with id 38819a34ca9a904cc34f79da9e956795288c136a2fe35db71272d2911fad764e
Jul 24 17:50:49.384111 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:49.384093    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:50:50.002456 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:50.002420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht" event={"ID":"a1b93aab-418a-45fe-9015-8200a69da666","Type":"ContainerStarted","Data":"38819a34ca9a904cc34f79da9e956795288c136a2fe35db71272d2911fad764e"}
Jul 24 17:50:51.006917 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:51.006885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht" event={"ID":"a1b93aab-418a-45fe-9015-8200a69da666","Type":"ContainerStarted","Data":"bcf8587f5da7f1e2188fca5230dbfb0a4503ad0559363e68a04c500a32daf4a8"}
Jul 24 17:50:51.035260 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:51.035210    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht" podStartSLOduration=2.04049191 podStartE2EDuration="3.03519393s" totalImagesPullingTime="994.70202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:50:51.033764207 +0000 UTC m=+864.550275244" watchObservedRunningTime="2026-07-24 17:50:51.03519393 +0000 UTC m=+864.551704966"
Jul 24 17:50:55.059775 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:55.059744    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:59.254241 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:59.254200    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:59.254684 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:59.254538    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:50:59.266481 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:50:59.266458    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:51:00.052772 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:00.052738    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht"
Jul 24 17:51:08.751855 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.751820    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"]
Jul 24 17:51:08.755065 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.755046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.757409 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.757391    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 24 17:51:08.765102 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.765081    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"]
Jul 24 17:51:08.861997 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.861968    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:08.897582 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.897542    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a776e008-7fad-4d1a-ab9d-913b48fff221-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.897711 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.897593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a776e008-7fad-4d1a-ab9d-913b48fff221-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.897711 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.897615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhdnn\" (UniqueName: \"kubernetes.io/projected/a776e008-7fad-4d1a-ab9d-913b48fff221-kube-api-access-rhdnn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.897711 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.897668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a776e008-7fad-4d1a-ab9d-913b48fff221-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.897813 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.897715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a776e008-7fad-4d1a-ab9d-913b48fff221-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.897813 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.897741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a776e008-7fad-4d1a-ab9d-913b48fff221-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.998370 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.998336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a776e008-7fad-4d1a-ab9d-913b48fff221-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.998543 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.998381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a776e008-7fad-4d1a-ab9d-913b48fff221-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.998543 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.998407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhdnn\" (UniqueName: \"kubernetes.io/projected/a776e008-7fad-4d1a-ab9d-913b48fff221-kube-api-access-rhdnn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.998543 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.998439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a776e008-7fad-4d1a-ab9d-913b48fff221-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.998543 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.998478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a776e008-7fad-4d1a-ab9d-913b48fff221-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.998543 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.998512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a776e008-7fad-4d1a-ab9d-913b48fff221-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.998828 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.998803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a776e008-7fad-4d1a-ab9d-913b48fff221-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.998894 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.998820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a776e008-7fad-4d1a-ab9d-913b48fff221-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:08.998894 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:08.998859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a776e008-7fad-4d1a-ab9d-913b48fff221-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:09.000703 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:09.000684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a776e008-7fad-4d1a-ab9d-913b48fff221-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:09.000971 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:09.000955    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a776e008-7fad-4d1a-ab9d-913b48fff221-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:09.005637 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:09.005590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhdnn\" (UniqueName: \"kubernetes.io/projected/a776e008-7fad-4d1a-ab9d-913b48fff221-kube-api-access-rhdnn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hnl8t\" (UID: \"a776e008-7fad-4d1a-ab9d-913b48fff221\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:09.064634 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:09.064610    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:09.188672 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:09.187970    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"]
Jul 24 17:51:10.089038 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:10.089001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t" event={"ID":"a776e008-7fad-4d1a-ab9d-913b48fff221","Type":"ContainerStarted","Data":"3fd56614e0d5cdca2ae2e09eed6cc23ef88931fd1f4f496b394b8b56513614ce"}
Jul 24 17:51:10.089477 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:10.089061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t" event={"ID":"a776e008-7fad-4d1a-ab9d-913b48fff221","Type":"ContainerStarted","Data":"03b9cea76580e1d468a316dee6beceb6c61a52dc53d7e0ca0ea6f4e2a6ebe2c8"}
Jul 24 17:51:10.106752 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:10.106702    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t" podStartSLOduration=1.861839888 podStartE2EDuration="2.106681432s" totalImagesPullingTime="244.841544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:51:10.104986293 +0000 UTC m=+883.621497326" watchObservedRunningTime="2026-07-24 17:51:10.106681432 +0000 UTC m=+883.623192471"
Jul 24 17:51:13.972109 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:13.972081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:19.065373 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:19.065290    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:19.065373 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:19.065356    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:19.077483 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:19.077457    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:19.136995 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:19.136971    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hnl8t"
Jul 24 17:51:20.498068 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:20.498035    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:28.861041 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:28.861004    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:32.163705 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:32.163669    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:32.277334 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:32.277301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:52.590037 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:52.590003    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:57.162559 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:51:57.162528    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:21.389947 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:52:21.389912    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:21.483161 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:52:21.483131    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:22.749168 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:52:22.749139    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:22.847994 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:52:22.847966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:36.976561 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:52:36.976526    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:37.144032 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:52:37.144003    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:24.103829 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:53:24.103755    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:41.771805 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:53:41.771772    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:12.189998 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.189963    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d4bb4965f-8kszx"]
Jul 24 17:54:12.193010 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.192986    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d4bb4965f-8kszx"
Jul 24 17:54:12.196343 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.196314    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 17:54:12.196343 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.196314    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-spv7j\""
Jul 24 17:54:12.196546 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.196316    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 17:54:12.203944 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.203922    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d4bb4965f-8kszx"]
Jul 24 17:54:12.300922 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.300885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td9tb\" (UniqueName: \"kubernetes.io/projected/e3ee7611-0e56-4d05-8fa6-b024c799be53-kube-api-access-td9tb\") pod \"maas-controller-d4bb4965f-8kszx\" (UID: \"e3ee7611-0e56-4d05-8fa6-b024c799be53\") " pod="opendatahub/maas-controller-d4bb4965f-8kszx"
Jul 24 17:54:12.301067 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.300954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ee7611-0e56-4d05-8fa6-b024c799be53-webhook-cert\") pod \"maas-controller-d4bb4965f-8kszx\" (UID: \"e3ee7611-0e56-4d05-8fa6-b024c799be53\") " pod="opendatahub/maas-controller-d4bb4965f-8kszx"
Jul 24 17:54:12.402201 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.402169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td9tb\" (UniqueName: \"kubernetes.io/projected/e3ee7611-0e56-4d05-8fa6-b024c799be53-kube-api-access-td9tb\") pod \"maas-controller-d4bb4965f-8kszx\" (UID: \"e3ee7611-0e56-4d05-8fa6-b024c799be53\") " pod="opendatahub/maas-controller-d4bb4965f-8kszx"
Jul 24 17:54:12.402306 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.402223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ee7611-0e56-4d05-8fa6-b024c799be53-webhook-cert\") pod \"maas-controller-d4bb4965f-8kszx\" (UID: \"e3ee7611-0e56-4d05-8fa6-b024c799be53\") " pod="opendatahub/maas-controller-d4bb4965f-8kszx"
Jul 24 17:54:12.404586 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.404556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ee7611-0e56-4d05-8fa6-b024c799be53-webhook-cert\") pod \"maas-controller-d4bb4965f-8kszx\" (UID: \"e3ee7611-0e56-4d05-8fa6-b024c799be53\") " pod="opendatahub/maas-controller-d4bb4965f-8kszx"
Jul 24 17:54:12.410704 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.410682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td9tb\" (UniqueName: \"kubernetes.io/projected/e3ee7611-0e56-4d05-8fa6-b024c799be53-kube-api-access-td9tb\") pod \"maas-controller-d4bb4965f-8kszx\" (UID: \"e3ee7611-0e56-4d05-8fa6-b024c799be53\") " pod="opendatahub/maas-controller-d4bb4965f-8kszx"
Jul 24 17:54:12.502464 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.502394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d4bb4965f-8kszx"
Jul 24 17:54:12.630790 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.630767    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d4bb4965f-8kszx"]
Jul 24 17:54:12.633264 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:54:12.633232    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ee7611_0e56_4d05_8fa6_b024c799be53.slice/crio-93e50de1d7369ae56c0dad01a1da76520dfd2b3adf852dc9550dd263882bbf1e WatchSource:0}: Error finding container 93e50de1d7369ae56c0dad01a1da76520dfd2b3adf852dc9550dd263882bbf1e: Status 404 returned error can't find the container with id 93e50de1d7369ae56c0dad01a1da76520dfd2b3adf852dc9550dd263882bbf1e
Jul 24 17:54:12.797339 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:12.797257    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d4bb4965f-8kszx" event={"ID":"e3ee7611-0e56-4d05-8fa6-b024c799be53","Type":"ContainerStarted","Data":"93e50de1d7369ae56c0dad01a1da76520dfd2b3adf852dc9550dd263882bbf1e"}
Jul 24 17:54:15.812613 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:15.812560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d4bb4965f-8kszx" event={"ID":"e3ee7611-0e56-4d05-8fa6-b024c799be53","Type":"ContainerStarted","Data":"ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27"}
Jul 24 17:54:15.813037 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:15.812807    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d4bb4965f-8kszx"
Jul 24 17:54:15.830826 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:15.830779    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d4bb4965f-8kszx" podStartSLOduration=1.424983858 podStartE2EDuration="3.830762254s" totalImagesPullingTime="2.405778396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:54:15.829893041 +0000 UTC m=+1069.346404081" watchObservedRunningTime="2026-07-24 17:54:15.830762254 +0000 UTC m=+1069.347273290"
Jul 24 17:54:26.819465 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:26.819439    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d4bb4965f-8kszx"
Jul 24 17:54:31.473691 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.473655    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d4bb4965f-8kszx"]
Jul 24 17:54:31.474092 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.473874    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d4bb4965f-8kszx" podUID="e3ee7611-0e56-4d05-8fa6-b024c799be53" containerName="manager" containerID="cri-o://ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27" gracePeriod=30
Jul 24 17:54:31.712913 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.712891    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d4bb4965f-8kszx"
Jul 24 17:54:31.856303 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.856277    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td9tb\" (UniqueName: \"kubernetes.io/projected/e3ee7611-0e56-4d05-8fa6-b024c799be53-kube-api-access-td9tb\") pod \"e3ee7611-0e56-4d05-8fa6-b024c799be53\" (UID: \"e3ee7611-0e56-4d05-8fa6-b024c799be53\") "
Jul 24 17:54:31.856466 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.856352    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ee7611-0e56-4d05-8fa6-b024c799be53-webhook-cert\") pod \"e3ee7611-0e56-4d05-8fa6-b024c799be53\" (UID: \"e3ee7611-0e56-4d05-8fa6-b024c799be53\") "
Jul 24 17:54:31.858494 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.858462    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ee7611-0e56-4d05-8fa6-b024c799be53-kube-api-access-td9tb" (OuterVolumeSpecName: "kube-api-access-td9tb") pod "e3ee7611-0e56-4d05-8fa6-b024c799be53" (UID: "e3ee7611-0e56-4d05-8fa6-b024c799be53"). InnerVolumeSpecName "kube-api-access-td9tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:54:31.858618 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.858560    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ee7611-0e56-4d05-8fa6-b024c799be53-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e3ee7611-0e56-4d05-8fa6-b024c799be53" (UID: "e3ee7611-0e56-4d05-8fa6-b024c799be53"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:54:31.876956 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.876930    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3ee7611-0e56-4d05-8fa6-b024c799be53" containerID="ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27" exitCode=0
Jul 24 17:54:31.877046 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.876974    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d4bb4965f-8kszx"
Jul 24 17:54:31.877046 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.877013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d4bb4965f-8kszx" event={"ID":"e3ee7611-0e56-4d05-8fa6-b024c799be53","Type":"ContainerDied","Data":"ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27"}
Jul 24 17:54:31.877118 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.877062    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d4bb4965f-8kszx" event={"ID":"e3ee7611-0e56-4d05-8fa6-b024c799be53","Type":"ContainerDied","Data":"93e50de1d7369ae56c0dad01a1da76520dfd2b3adf852dc9550dd263882bbf1e"}
Jul 24 17:54:31.877118 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.877086    2581 scope.go:117] "RemoveContainer" containerID="ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27"
Jul 24 17:54:31.885038 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.885017    2581 scope.go:117] "RemoveContainer" containerID="ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27"
Jul 24 17:54:31.885287 ip-10-0-141-44 kubenswrapper[2581]: E0724 17:54:31.885265    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27\": container with ID starting with ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27 not found: ID does not exist" containerID="ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27"
Jul 24 17:54:31.885361 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.885297    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27"} err="failed to get container status \"ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27\": rpc error: code = NotFound desc = could not find container \"ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27\": container with ID starting with ae85e3505e787a911197bda6ea741353a1e40c8a93ad1740e83ee2fe8202ea27 not found: ID does not exist"
Jul 24 17:54:31.899058 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.899038    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d4bb4965f-8kszx"]
Jul 24 17:54:31.903452 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.903429    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d4bb4965f-8kszx"]
Jul 24 17:54:31.957147 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.957126    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td9tb\" (UniqueName: \"kubernetes.io/projected/e3ee7611-0e56-4d05-8fa6-b024c799be53-kube-api-access-td9tb\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 17:54:31.957229 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.957148    2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ee7611-0e56-4d05-8fa6-b024c799be53-webhook-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 17:54:31.972800 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:31.972779    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:32.604513 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.604476    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:32.719824 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.719793    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d4bb4965f-bhxsx"]
Jul 24 17:54:32.720086 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.720072    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ee7611-0e56-4d05-8fa6-b024c799be53" containerName="manager"
Jul 24 17:54:32.720131 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.720088    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ee7611-0e56-4d05-8fa6-b024c799be53" containerName="manager"
Jul 24 17:54:32.720165 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.720153    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ee7611-0e56-4d05-8fa6-b024c799be53" containerName="manager"
Jul 24 17:54:32.724219 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.724201    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d4bb4965f-bhxsx"
Jul 24 17:54:32.727478 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.727460    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-spv7j\""
Jul 24 17:54:32.727604 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.727475    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 17:54:32.727604 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.727538    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 17:54:32.732525 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.732504    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d4bb4965f-bhxsx"]
Jul 24 17:54:32.863796 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.863717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8bfbfd3-e3ad-4e7c-a922-f0e97d337188-webhook-cert\") pod \"maas-controller-d4bb4965f-bhxsx\" (UID: \"b8bfbfd3-e3ad-4e7c-a922-f0e97d337188\") " pod="opendatahub/maas-controller-d4bb4965f-bhxsx"
Jul 24 17:54:32.863927 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.863798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6c4t\" (UniqueName: \"kubernetes.io/projected/b8bfbfd3-e3ad-4e7c-a922-f0e97d337188-kube-api-access-d6c4t\") pod \"maas-controller-d4bb4965f-bhxsx\" (UID: \"b8bfbfd3-e3ad-4e7c-a922-f0e97d337188\") " pod="opendatahub/maas-controller-d4bb4965f-bhxsx"
Jul 24 17:54:32.957340 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.957303    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3ee7611-0e56-4d05-8fa6-b024c799be53" path="/var/lib/kubelet/pods/e3ee7611-0e56-4d05-8fa6-b024c799be53/volumes"
Jul 24 17:54:32.964677 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.964659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6c4t\" (UniqueName: \"kubernetes.io/projected/b8bfbfd3-e3ad-4e7c-a922-f0e97d337188-kube-api-access-d6c4t\") pod \"maas-controller-d4bb4965f-bhxsx\" (UID: \"b8bfbfd3-e3ad-4e7c-a922-f0e97d337188\") " pod="opendatahub/maas-controller-d4bb4965f-bhxsx"
Jul 24 17:54:32.964767 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.964696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8bfbfd3-e3ad-4e7c-a922-f0e97d337188-webhook-cert\") pod \"maas-controller-d4bb4965f-bhxsx\" (UID: \"b8bfbfd3-e3ad-4e7c-a922-f0e97d337188\") " pod="opendatahub/maas-controller-d4bb4965f-bhxsx"
Jul 24 17:54:32.967054 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.967033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8bfbfd3-e3ad-4e7c-a922-f0e97d337188-webhook-cert\") pod \"maas-controller-d4bb4965f-bhxsx\" (UID: \"b8bfbfd3-e3ad-4e7c-a922-f0e97d337188\") " pod="opendatahub/maas-controller-d4bb4965f-bhxsx"
Jul 24 17:54:32.974652 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:32.974626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6c4t\" (UniqueName: \"kubernetes.io/projected/b8bfbfd3-e3ad-4e7c-a922-f0e97d337188-kube-api-access-d6c4t\") pod \"maas-controller-d4bb4965f-bhxsx\" (UID: \"b8bfbfd3-e3ad-4e7c-a922-f0e97d337188\") " pod="opendatahub/maas-controller-d4bb4965f-bhxsx"
Jul 24 17:54:33.034612 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:33.034556    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d4bb4965f-bhxsx"
Jul 24 17:54:33.159697 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:33.159671    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d4bb4965f-bhxsx"]
Jul 24 17:54:33.162236 ip-10-0-141-44 kubenswrapper[2581]: W0724 17:54:33.162210    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8bfbfd3_e3ad_4e7c_a922_f0e97d337188.slice/crio-d0e0ce2867a16fc45c1e4ebbb3c523f38bd43fcb3fae8f2dc95dbe253de1e13d WatchSource:0}: Error finding container d0e0ce2867a16fc45c1e4ebbb3c523f38bd43fcb3fae8f2dc95dbe253de1e13d: Status 404 returned error can't find the container with id d0e0ce2867a16fc45c1e4ebbb3c523f38bd43fcb3fae8f2dc95dbe253de1e13d
Jul 24 17:54:33.888005 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:33.887972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d4bb4965f-bhxsx" event={"ID":"b8bfbfd3-e3ad-4e7c-a922-f0e97d337188","Type":"ContainerStarted","Data":"b79ec360ffe3a38ea7bd6fdb86bdb7367901f92192d323d1299ddc539f015e3d"}
Jul 24 17:54:33.888389 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:33.888026    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d4bb4965f-bhxsx" event={"ID":"b8bfbfd3-e3ad-4e7c-a922-f0e97d337188","Type":"ContainerStarted","Data":"d0e0ce2867a16fc45c1e4ebbb3c523f38bd43fcb3fae8f2dc95dbe253de1e13d"}
Jul 24 17:54:33.888389 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:33.888059    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d4bb4965f-bhxsx"
Jul 24 17:54:33.906542 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:33.906501    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d4bb4965f-bhxsx" podStartSLOduration=1.595505793 podStartE2EDuration="1.906486099s" totalImagesPullingTime="310.980306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:54:33.9049186 +0000 UTC m=+1087.421429636" watchObservedRunningTime="2026-07-24 17:54:33.906486099 +0000 UTC m=+1087.422997135"
Jul 24 17:54:44.894521 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:54:44.894493    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d4bb4965f-bhxsx"
Jul 24 17:55:10.258099 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:55:10.258057    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:10.357641 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:55:10.357614    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:10.463789 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:55:10.463755    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:16.686053 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:55:16.686019    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:16.787212 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:55:16.787179    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:16.886751 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:55:16.886719    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:17.044712 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:55:17.044633    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:22.732564 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:55:22.732525    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:22.809680 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:55:22.809645    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:51.343320 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:56:51.343278    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:59:15.207984 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:59:15.207887    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:59:22.270258 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:59:22.270224    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 17:59:22.365320 ip-10-0-141-44 kubenswrapper[2581]: I0724 17:59:22.365279    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:00:31.801803 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:00:31.801750    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:03:03.246590 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:03:03.246527    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:03:18.876926 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:03:18.876893    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:03:45.488280 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:03:45.488248    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:03:46.575345 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:03:46.575309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:04:20.379844 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:04:20.379812    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:04:22.044209 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:04:22.044174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:04:36.261623 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:04:36.261589    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:04:36.330459 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:04:36.330428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:06:57.492265 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:06:57.492233    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:01.274708 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:01.274663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:22.850601 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:22.850548    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:26.162144 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:26.162113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:26.257604 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:26.257551    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:26.691913 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:26.691877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:32.514948 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:32.514914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:32.615261 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:32.615228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:32.797557 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:32.797472    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:33.090221 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.090190    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"]
Jul 24 18:07:33.093297 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.093275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.095697 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.095680    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d49fe598-openshift-default-dockercfg-9jbmf\""
Jul 24 18:07:33.114654 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.114634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"]
Jul 24 18:07:33.117485 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.117458    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:33.153631 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.153604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a491614f-c757-466d-9ccc-ce849610c87b-istio-podinfo\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.153726 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.153646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-workload-socket\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.153726 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.153661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-istio-envoy\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.153726 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.153702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a491614f-c757-466d-9ccc-ce849610c87b-istiod-ca-cert\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.153828 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.153736    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-credential-socket\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.153828 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.153756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcqhh\" (UniqueName: \"kubernetes.io/projected/a491614f-c757-466d-9ccc-ce849610c87b-kube-api-access-mcqhh\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.153828 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.153814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-workload-certs\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.153919 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.153845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a491614f-c757-466d-9ccc-ce849610c87b-istio-token\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.153919 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.153862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-istio-data\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255036 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.254999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a491614f-c757-466d-9ccc-ce849610c87b-istio-token\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255036 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-istio-data\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255258 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a491614f-c757-466d-9ccc-ce849610c87b-istio-podinfo\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255258 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-workload-socket\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255258 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-istio-envoy\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255258 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a491614f-c757-466d-9ccc-ce849610c87b-istiod-ca-cert\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255258 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-credential-socket\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255258 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcqhh\" (UniqueName: \"kubernetes.io/projected/a491614f-c757-466d-9ccc-ce849610c87b-kube-api-access-mcqhh\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255550 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-workload-certs\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255550 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255488    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-istio-data\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255550 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-workload-socket\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255740 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-workload-certs\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-credential-socket\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.255998 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.255980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a491614f-c757-466d-9ccc-ce849610c87b-istiod-ca-cert\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.257453 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.257431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-istio-envoy\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.257669 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.257652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a491614f-c757-466d-9ccc-ce849610c87b-istio-podinfo\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.263168 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.263145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a491614f-c757-466d-9ccc-ce849610c87b-istio-token\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.263397 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.263380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcqhh\" (UniqueName: \"kubernetes.io/projected/a491614f-c757-466d-9ccc-ce849610c87b-kube-api-access-mcqhh\") pod \"e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") " pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.392978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.392893    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:33.402853 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.402831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:33.532432 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.532378    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"]
Jul 24 18:07:33.535015 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:07:33.534981    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda491614f_c757_466d_9ccc_ce849610c87b.slice/crio-c4a5755d4f93b2e382db42c5f1b0dc615c5adc208a3d12940c198f4e6740f37e WatchSource:0}: Error finding container c4a5755d4f93b2e382db42c5f1b0dc615c5adc208a3d12940c198f4e6740f37e: Status 404 returned error can't find the container with id c4a5755d4f93b2e382db42c5f1b0dc615c5adc208a3d12940c198f4e6740f37e
Jul 24 18:07:33.537003 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.536985    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:07:33.970618 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:33.970563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz" event={"ID":"a491614f-c757-466d-9ccc-ce849610c87b","Type":"ContainerStarted","Data":"c4a5755d4f93b2e382db42c5f1b0dc615c5adc208a3d12940c198f4e6740f37e"}
Jul 24 18:07:36.220354 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:36.220322    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:07:36.220605 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:36.220398    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:07:36.220605 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:36.220433    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:07:36.986073 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:36.986044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz" event={"ID":"a491614f-c757-466d-9ccc-ce849610c87b","Type":"ContainerStarted","Data":"006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c"}
Jul 24 18:07:37.007763 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:37.007715    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz" podStartSLOduration=1.324771592 podStartE2EDuration="4.00769948s" totalImagesPullingTime="2.682927888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:07:37.007010517 +0000 UTC m=+1870.523521554" watchObservedRunningTime="2026-07-24 18:07:37.00769948 +0000 UTC m=+1870.524210515"
Jul 24 18:07:37.403229 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:37.403198    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:37.408065 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:37.408042    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:37.990180 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:37.990150    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:37.991122 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:37.991097    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:38.369081 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.369039    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:38.463695 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.463660    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:38.599254 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.599220    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"]
Jul 24 18:07:38.603090 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.603067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:38.605547 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.605514    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:07:38.605778 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.605562    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:07:38.606321 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.606299    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fvm85\""
Jul 24 18:07:38.606448 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.606301    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d49fe598-serving-cert\""
Jul 24 18:07:38.611212 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.611187    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"]
Jul 24 18:07:38.633822 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.633758    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:38.705239 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.705206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-openshift-service-ca\") pod \"maas-api-e2e-ait-d49fe598-c599484b-gm65b\" (UID: \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:38.705239 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.705243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtrmm\" (UniqueName: \"kubernetes.io/projected/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-kube-api-access-jtrmm\") pod \"maas-api-e2e-ait-d49fe598-c599484b-gm65b\" (UID: \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:38.705427 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.705275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-maas-api-tls\") pod \"maas-api-e2e-ait-d49fe598-c599484b-gm65b\" (UID: \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:38.732804 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.732776    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:38.805979 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.805949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-openshift-service-ca\") pod \"maas-api-e2e-ait-d49fe598-c599484b-gm65b\" (UID: \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:38.806137 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.805983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtrmm\" (UniqueName: \"kubernetes.io/projected/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-kube-api-access-jtrmm\") pod \"maas-api-e2e-ait-d49fe598-c599484b-gm65b\" (UID: \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:38.806137 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.806009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-maas-api-tls\") pod \"maas-api-e2e-ait-d49fe598-c599484b-gm65b\" (UID: \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:38.806663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.806638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-openshift-service-ca\") pod \"maas-api-e2e-ait-d49fe598-c599484b-gm65b\" (UID: \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:38.808905 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.808867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-maas-api-tls\") pod \"maas-api-e2e-ait-d49fe598-c599484b-gm65b\" (UID: \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:38.817298 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.817249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtrmm\" (UniqueName: \"kubernetes.io/projected/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-kube-api-access-jtrmm\") pod \"maas-api-e2e-ait-d49fe598-c599484b-gm65b\" (UID: \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:38.915092 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:38.915018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:39.048004 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.047979    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"]
Jul 24 18:07:39.050434 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:07:39.050412    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199469a3_e6e6_44ea_b4a4_61f9c7880a4e.slice/crio-06d58c7004a5a683e152ea3745f0c4afed322743878e8ea0fff49844c9f36687 WatchSource:0}: Error finding container 06d58c7004a5a683e152ea3745f0c4afed322743878e8ea0fff49844c9f36687: Status 404 returned error can't find the container with id 06d58c7004a5a683e152ea3745f0c4afed322743878e8ea0fff49844c9f36687
Jul 24 18:07:39.126167 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.126137    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:39.225070 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.224987    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:07:39.242219 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.242188    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"]
Jul 24 18:07:39.246537 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.246516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"
Jul 24 18:07:39.249899 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.249876    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d49fe598\""
Jul 24 18:07:39.259739 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.259716    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"]
Jul 24 18:07:39.311528 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.311492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj\" (UID: \"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"
Jul 24 18:07:39.311528 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.311527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t68gt\" (UniqueName: \"kubernetes.io/projected/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f-kube-api-access-t68gt\") pod \"payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj\" (UID: \"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"
Jul 24 18:07:39.412302 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.412271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj\" (UID: \"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"
Jul 24 18:07:39.412302 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.412305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t68gt\" (UniqueName: \"kubernetes.io/projected/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f-kube-api-access-t68gt\") pod \"payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj\" (UID: \"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"
Jul 24 18:07:39.413059 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.413039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj\" (UID: \"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"
Jul 24 18:07:39.421261 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.421231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t68gt\" (UniqueName: \"kubernetes.io/projected/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f-kube-api-access-t68gt\") pod \"payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj\" (UID: \"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"
Jul 24 18:07:39.557221 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.557197    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"
Jul 24 18:07:39.686759 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.686722    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"]
Jul 24 18:07:39.689811 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:07:39.689784    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ff2c6e_6d88_4f88_af02_c6bcb8c7ec7f.slice/crio-757ecdae37594f2da8191c33100c18d1f09469b025b67a0de713a2a852d0fcea WatchSource:0}: Error finding container 757ecdae37594f2da8191c33100c18d1f09469b025b67a0de713a2a852d0fcea: Status 404 returned error can't find the container with id 757ecdae37594f2da8191c33100c18d1f09469b025b67a0de713a2a852d0fcea
Jul 24 18:07:39.999903 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:39.999817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj" event={"ID":"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f","Type":"ContainerStarted","Data":"757ecdae37594f2da8191c33100c18d1f09469b025b67a0de713a2a852d0fcea"}
Jul 24 18:07:40.000917 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:40.000891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b" event={"ID":"199469a3-e6e6-44ea-b4a4-61f9c7880a4e","Type":"ContainerStarted","Data":"06d58c7004a5a683e152ea3745f0c4afed322743878e8ea0fff49844c9f36687"}
Jul 24 18:07:41.008799 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:41.008667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b" event={"ID":"199469a3-e6e6-44ea-b4a4-61f9c7880a4e","Type":"ContainerStarted","Data":"31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2"}
Jul 24 18:07:41.009232 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:41.008864    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:41.032232 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:41.032168    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b" podStartSLOduration=1.369782599 podStartE2EDuration="3.032151692s" totalImagesPullingTime="1.662369093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:07:41.031034217 +0000 UTC m=+1874.547545268" watchObservedRunningTime="2026-07-24 18:07:41.032151692 +0000 UTC m=+1874.548662728"
Jul 24 18:07:42.014042 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:42.013997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj" event={"ID":"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f","Type":"ContainerStarted","Data":"bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423"}
Jul 24 18:07:42.014421 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:42.014089    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"
Jul 24 18:07:42.030934 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:42.030887    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj" podStartSLOduration=0.93744675 podStartE2EDuration="3.0308744s" totalImagesPullingTime="2.09342765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:07:42.029436745 +0000 UTC m=+1875.545947793" watchObservedRunningTime="2026-07-24 18:07:42.0308744 +0000 UTC m=+1875.547385436"
Jul 24 18:07:47.019454 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:47.019429    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:07:50.048970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:50.048890    2581 generic.go:356] "Generic (PLEG): container finished" podID="a491614f-c757-466d-9ccc-ce849610c87b" containerID="006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c" exitCode=0
Jul 24 18:07:50.048970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:50.048959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz" event={"ID":"a491614f-c757-466d-9ccc-ce849610c87b","Type":"ContainerDied","Data":"006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c"}
Jul 24 18:07:50.049407 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:50.049336    2581 scope.go:117] "RemoveContainer" containerID="006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c"
Jul 24 18:07:50.051772 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:50.051733    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:07:50.051888 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:50.051827    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:07:50.051888 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:50.051876    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:07:51.058755 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:51.058716    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz" event={"ID":"a491614f-c757-466d-9ccc-ce849610c87b","Type":"ContainerStarted","Data":"b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175"}
Jul 24 18:07:51.403400 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:51.403303    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:51.404436 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:51.404407    2581 patch_prober.go:28] interesting pod/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body=
Jul 24 18:07:51.404548 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:51.404459    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz" podUID="a491614f-c757-466d-9ccc-ce849610c87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused"
Jul 24 18:07:52.404204 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:52.404167    2581 patch_prober.go:28] interesting pod/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body=
Jul 24 18:07:52.404606 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:52.404223    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz" podUID="a491614f-c757-466d-9ccc-ce849610c87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused"
Jul 24 18:07:52.990896 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:52.990848    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:53.019509 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:53.019478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"
Jul 24 18:07:53.403655 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:53.403615    2581 patch_prober.go:28] interesting pod/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body=
Jul 24 18:07:53.403832 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:53.403683    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz" podUID="a491614f-c757-466d-9ccc-ce849610c87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused"
Jul 24 18:07:54.403844 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:54.403804    2581 patch_prober.go:28] interesting pod/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body=
Jul 24 18:07:54.404227 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:54.403864    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz" podUID="a491614f-c757-466d-9ccc-ce849610c87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused"
Jul 24 18:07:55.407688 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:55.407659    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:07:55.408732 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:07:55.408713    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:08:14.601647 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:14.601616    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"]
Jul 24 18:08:14.602155 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:14.601816    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b" podUID="199469a3-e6e6-44ea-b4a4-61f9c7880a4e" containerName="maas-api" containerID="cri-o://31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2" gracePeriod=30
Jul 24 18:08:14.621945 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:14.621869    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:14.658000 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:14.657971    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"]
Jul 24 18:08:14.658233 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:14.658210    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj" podUID="66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f" containerName="payload-pre-processing" containerID="cri-o://bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423" gracePeriod=30
Jul 24 18:08:14.720478 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:14.720451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:14.822077 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:14.822044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:14.925250 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:14.925229    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:08:14.927934 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:14.927915    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"
Jul 24 18:08:15.115216 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.115131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t68gt\" (UniqueName: \"kubernetes.io/projected/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f-kube-api-access-t68gt\") pod \"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f\" (UID: \"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f\") "
Jul 24 18:08:15.115216 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.115179    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtrmm\" (UniqueName: \"kubernetes.io/projected/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-kube-api-access-jtrmm\") pod \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\" (UID: \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\") "
Jul 24 18:08:15.115408 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.115223    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-openshift-service-ca\") pod \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\" (UID: \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\") "
Jul 24 18:08:15.115408 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.115247    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-maas-api-tls\") pod \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\" (UID: \"199469a3-e6e6-44ea-b4a4-61f9c7880a4e\") "
Jul 24 18:08:15.115408 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.115295    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f-plugins-config-volume\") pod \"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f\" (UID: \"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f\") "
Jul 24 18:08:15.115673 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.115641    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "199469a3-e6e6-44ea-b4a4-61f9c7880a4e" (UID: "199469a3-e6e6-44ea-b4a4-61f9c7880a4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:08:15.115780 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.115754    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f" (UID: "66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:08:15.117476 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.117451    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "199469a3-e6e6-44ea-b4a4-61f9c7880a4e" (UID: "199469a3-e6e6-44ea-b4a4-61f9c7880a4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:08:15.117534 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.117459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-kube-api-access-jtrmm" (OuterVolumeSpecName: "kube-api-access-jtrmm") pod "199469a3-e6e6-44ea-b4a4-61f9c7880a4e" (UID: "199469a3-e6e6-44ea-b4a4-61f9c7880a4e"). InnerVolumeSpecName "kube-api-access-jtrmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:08:15.117534 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.117493    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f-kube-api-access-t68gt" (OuterVolumeSpecName: "kube-api-access-t68gt") pod "66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f" (UID: "66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f"). InnerVolumeSpecName "kube-api-access-t68gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:08:15.164461 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.164424    2581 generic.go:356] "Generic (PLEG): container finished" podID="66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f" containerID="bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423" exitCode=0
Jul 24 18:08:15.164620 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.164476    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"
Jul 24 18:08:15.164620 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.164497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj" event={"ID":"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f","Type":"ContainerDied","Data":"bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423"}
Jul 24 18:08:15.164620 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.164551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj" event={"ID":"66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f","Type":"ContainerDied","Data":"757ecdae37594f2da8191c33100c18d1f09469b025b67a0de713a2a852d0fcea"}
Jul 24 18:08:15.164620 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.164589    2581 scope.go:117] "RemoveContainer" containerID="bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423"
Jul 24 18:08:15.165985 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.165936    2581 generic.go:356] "Generic (PLEG): container finished" podID="199469a3-e6e6-44ea-b4a4-61f9c7880a4e" containerID="31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2" exitCode=0
Jul 24 18:08:15.165985 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.165971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b" event={"ID":"199469a3-e6e6-44ea-b4a4-61f9c7880a4e","Type":"ContainerDied","Data":"31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2"}
Jul 24 18:08:15.166217 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.166001    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"
Jul 24 18:08:15.166217 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.166032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b" event={"ID":"199469a3-e6e6-44ea-b4a4-61f9c7880a4e","Type":"ContainerDied","Data":"06d58c7004a5a683e152ea3745f0c4afed322743878e8ea0fff49844c9f36687"}
Jul 24 18:08:15.174544 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.174527    2581 scope.go:117] "RemoveContainer" containerID="bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423"
Jul 24 18:08:15.174905 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:08:15.174803    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423\": container with ID starting with bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423 not found: ID does not exist" containerID="bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423"
Jul 24 18:08:15.174905 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.174828    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423"} err="failed to get container status \"bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423\": rpc error: code = NotFound desc = could not find container \"bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423\": container with ID starting with bad58aef0b400e9586a70462e969cda6f4dca844935089e759e8c5a7cbf32423 not found: ID does not exist"
Jul 24 18:08:15.174905 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.174845    2581 scope.go:117] "RemoveContainer" containerID="31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2"
Jul 24 18:08:15.181889 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.181873    2581 scope.go:117] "RemoveContainer" containerID="31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2"
Jul 24 18:08:15.182144 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:08:15.182126    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2\": container with ID starting with 31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2 not found: ID does not exist" containerID="31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2"
Jul 24 18:08:15.182192 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.182149    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2"} err="failed to get container status \"31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2\": rpc error: code = NotFound desc = could not find container \"31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2\": container with ID starting with 31e7da4cfd41fa71dcd94bbe24b5dd5e1a8eb5aee2ff9647a7a3a9d77d33e7f2 not found: ID does not exist"
Jul 24 18:08:15.190180 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.190161    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"]
Jul 24 18:08:15.194358 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.194339    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d49fe598-c599484b-gm65b"]
Jul 24 18:08:15.204345 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.204327    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"]
Jul 24 18:08:15.207915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.207897    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d49fe598-ff7859b7b-gldrj"]
Jul 24 18:08:15.216071 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.216051    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:15.216163 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.216076    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:15.216163 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.216091    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:15.216163 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.216105    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t68gt\" (UniqueName: \"kubernetes.io/projected/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f-kube-api-access-t68gt\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:15.216163 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:15.216119    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtrmm\" (UniqueName: \"kubernetes.io/projected/199469a3-e6e6-44ea-b4a4-61f9c7880a4e-kube-api-access-jtrmm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:16.958728 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:16.958695    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199469a3-e6e6-44ea-b4a4-61f9c7880a4e" path="/var/lib/kubelet/pods/199469a3-e6e6-44ea-b4a4-61f9c7880a4e/volumes"
Jul 24 18:08:16.959097 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:16.959038    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f" path="/var/lib/kubelet/pods/66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f/volumes"
Jul 24 18:08:19.077131 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:19.077098    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:19.217014 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:19.216984    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"]
Jul 24 18:08:19.217248 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:19.217209    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz" podUID="a491614f-c757-466d-9ccc-ce849610c87b" containerName="istio-proxy" containerID="cri-o://b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175" gracePeriod=30
Jul 24 18:08:19.237194 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:19.237164    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:19.486749 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:19.486646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:19.755957 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:19.755883    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:19.855847 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:19.855811    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:20.000321 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.000283    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:20.323589 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.323546    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"]
Jul 24 18:08:20.324006 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.323868    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199469a3-e6e6-44ea-b4a4-61f9c7880a4e" containerName="maas-api"
Jul 24 18:08:20.324006 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.323880    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="199469a3-e6e6-44ea-b4a4-61f9c7880a4e" containerName="maas-api"
Jul 24 18:08:20.324006 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.323894    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f" containerName="payload-pre-processing"
Jul 24 18:08:20.324006 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.323901    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f" containerName="payload-pre-processing"
Jul 24 18:08:20.324006 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.323954    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="199469a3-e6e6-44ea-b4a4-61f9c7880a4e" containerName="maas-api"
Jul 24 18:08:20.324006 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.323962    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ff2c6e-6d88-4f88-af02-c6bcb8c7ec7f" containerName="payload-pre-processing"
Jul 24 18:08:20.328409 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.328388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.330967 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.330944    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-23b920dc-gw-openshift-default-dockercfg-tgcvb\""
Jul 24 18:08:20.341159 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.341137    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"]
Jul 24 18:08:20.352213 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.352189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-workload-certs\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.352326 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.352217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-token\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.352326 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.352243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-data\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.352445 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.352349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4slt\" (UniqueName: \"kubernetes.io/projected/70b84616-1fec-4e79-a32a-ff4ee81b994d-kube-api-access-s4slt\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.352445 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.352392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-workload-socket\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.352445 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.352432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-credential-socket\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.352548 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.352489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b84616-1fec-4e79-a32a-ff4ee81b994d-istiod-ca-cert\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.352548 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.352515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-envoy\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.352548 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.352532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-podinfo\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.354546 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.354523    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:20.453020 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.452986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-workload-certs\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.453184 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.453034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-token\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.453184 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.453065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-data\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.453184 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.453107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4slt\" (UniqueName: \"kubernetes.io/projected/70b84616-1fec-4e79-a32a-ff4ee81b994d-kube-api-access-s4slt\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.453184 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.453140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-workload-socket\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.453184 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.453176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-credential-socket\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.453468 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.453219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b84616-1fec-4e79-a32a-ff4ee81b994d-istiod-ca-cert\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.453468 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.453238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-envoy\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.453468 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.453254    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-podinfo\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.453468 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.453400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-workload-certs\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.453760 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.453743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-workload-socket\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.453821 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.453763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-data\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.453821 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.453802    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:20.454064 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.454040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-credential-socket\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.454188 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.454171    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b84616-1fec-4e79-a32a-ff4ee81b994d-istiod-ca-cert\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.455741 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.455721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-envoy\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.455837 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.455759    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-podinfo\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.461143 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.461111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-token\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.461251 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.461226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4slt\" (UniqueName: \"kubernetes.io/projected/70b84616-1fec-4e79-a32a-ff4ee81b994d-kube-api-access-s4slt\") pod \"e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") " pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.639256 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.639176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:20.785646 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:08:20.785615    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b84616_1fec_4e79_a32a_ff4ee81b994d.slice/crio-ba393ac801a0b861c40e83f04125ea93978057c4e6416c77db72692ff0cc06fd WatchSource:0}: Error finding container ba393ac801a0b861c40e83f04125ea93978057c4e6416c77db72692ff0cc06fd: Status 404 returned error can't find the container with id ba393ac801a0b861c40e83f04125ea93978057c4e6416c77db72692ff0cc06fd
Jul 24 18:08:20.788538 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.788503    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:08:20.788645 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.788596    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:08:20.788645 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.788624    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:08:20.789953 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:20.789931    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"]
Jul 24 18:08:21.194835 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:21.194801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz" event={"ID":"70b84616-1fec-4e79-a32a-ff4ee81b994d","Type":"ContainerStarted","Data":"f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6"}
Jul 24 18:08:21.195022 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:21.194850    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz" event={"ID":"70b84616-1fec-4e79-a32a-ff4ee81b994d","Type":"ContainerStarted","Data":"ba393ac801a0b861c40e83f04125ea93978057c4e6416c77db72692ff0cc06fd"}
Jul 24 18:08:21.215455 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:21.215401    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz" podStartSLOduration=1.215385152 podStartE2EDuration="1.215385152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:08:21.21381075 +0000 UTC m=+1914.730321783" watchObservedRunningTime="2026-07-24 18:08:21.215385152 +0000 UTC m=+1914.731896187"
Jul 24 18:08:21.639760 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:21.639728    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:21.645254 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:21.645228    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:22.199124 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.199088    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:22.200117 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.200100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:22.687372 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.687238    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:22.793778 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.793738    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"]
Jul 24 18:08:22.797012 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.796988    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:22.799266 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.799241    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-23b920dc-serving-cert\""
Jul 24 18:08:22.799369 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.799352    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:08:22.799436 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.799395    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fvm85\""
Jul 24 18:08:22.800252 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.800230    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:08:22.806321 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.806300    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"]
Jul 24 18:08:22.874879 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.874846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-openshift-service-ca\") pod \"maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:22.875017 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.874886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-maas-api-tls\") pod \"maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:22.875017 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.874907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqjj\" (UniqueName: \"kubernetes.io/projected/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-kube-api-access-2xqjj\") pod \"maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:22.897759 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.897728    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:22.976389 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.976301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-openshift-service-ca\") pod \"maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:22.976389 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.976356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-maas-api-tls\") pod \"maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:22.976389 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.976383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqjj\" (UniqueName: \"kubernetes.io/projected/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-kube-api-access-2xqjj\") pod \"maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:22.976701 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:08:22.976524    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-serving-cert: secret "maas-api-e2e-migrate-23b920dc-serving-cert" not found
Jul 24 18:08:22.976701 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:08:22.976625    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-maas-api-tls podName:4d6c65d7-75b9-4c97-806c-9c89e9d812bb nodeName:}" failed. No retries permitted until 2026-07-24 18:08:23.476603653 +0000 UTC m=+1916.993114682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-maas-api-tls") pod "maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2" (UID: "4d6c65d7-75b9-4c97-806c-9c89e9d812bb") : secret "maas-api-e2e-migrate-23b920dc-serving-cert" not found
Jul 24 18:08:22.977061 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.977043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-openshift-service-ca\") pod \"maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:22.985481 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:22.985464    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqjj\" (UniqueName: \"kubernetes.io/projected/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-kube-api-access-2xqjj\") pod \"maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:23.139795 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.139762    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:23.352947 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.352913    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:23.441200 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.441140    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:23.445439 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.445412    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"]
Jul 24 18:08:23.448639 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.448624    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"
Jul 24 18:08:23.451454 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.451434    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-23b920dc\""
Jul 24 18:08:23.460015 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.459996    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"]
Jul 24 18:08:23.479986 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.479958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cchc\" (UniqueName: \"kubernetes.io/projected/674a0404-56a2-49b3-967e-706f12b5cff3-kube-api-access-2cchc\") pod \"payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g\" (UID: \"674a0404-56a2-49b3-967e-706f12b5cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"
Jul 24 18:08:23.480117 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.480007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/674a0404-56a2-49b3-967e-706f12b5cff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g\" (UID: \"674a0404-56a2-49b3-967e-706f12b5cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"
Jul 24 18:08:23.480117 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.480040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-maas-api-tls\") pod \"maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:23.482611 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.482591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-maas-api-tls\") pod \"maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:23.580998 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.580962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/674a0404-56a2-49b3-967e-706f12b5cff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g\" (UID: \"674a0404-56a2-49b3-967e-706f12b5cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"
Jul 24 18:08:23.581165 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.581062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cchc\" (UniqueName: \"kubernetes.io/projected/674a0404-56a2-49b3-967e-706f12b5cff3-kube-api-access-2cchc\") pod \"payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g\" (UID: \"674a0404-56a2-49b3-967e-706f12b5cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"
Jul 24 18:08:23.581707 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.581689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/674a0404-56a2-49b3-967e-706f12b5cff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g\" (UID: \"674a0404-56a2-49b3-967e-706f12b5cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"
Jul 24 18:08:23.588912 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.588889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cchc\" (UniqueName: \"kubernetes.io/projected/674a0404-56a2-49b3-967e-706f12b5cff3-kube-api-access-2cchc\") pod \"payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g\" (UID: \"674a0404-56a2-49b3-967e-706f12b5cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"
Jul 24 18:08:23.707548 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.707458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:23.759675 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.759466    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"
Jul 24 18:08:23.863182 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.863159    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"]
Jul 24 18:08:23.865603 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:08:23.865534    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6c65d7_75b9_4c97_806c_9c89e9d812bb.slice/crio-daf8e9744af9d9885c25660089e5fe0fa2fcbba8fcd65956df3136f6d989ed43 WatchSource:0}: Error finding container daf8e9744af9d9885c25660089e5fe0fa2fcbba8fcd65956df3136f6d989ed43: Status 404 returned error can't find the container with id daf8e9744af9d9885c25660089e5fe0fa2fcbba8fcd65956df3136f6d989ed43
Jul 24 18:08:23.926198 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:23.926169    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"]
Jul 24 18:08:23.928657 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:08:23.928619    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674a0404_56a2_49b3_967e_706f12b5cff3.slice/crio-9be952f955aa627995397c5f6d0db9eb5965e83950079884886abef382fd679b WatchSource:0}: Error finding container 9be952f955aa627995397c5f6d0db9eb5965e83950079884886abef382fd679b: Status 404 returned error can't find the container with id 9be952f955aa627995397c5f6d0db9eb5965e83950079884886abef382fd679b
Jul 24 18:08:24.208360 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.208327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g" event={"ID":"674a0404-56a2-49b3-967e-706f12b5cff3","Type":"ContainerStarted","Data":"20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c"}
Jul 24 18:08:24.208536 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.208377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g" event={"ID":"674a0404-56a2-49b3-967e-706f12b5cff3","Type":"ContainerStarted","Data":"9be952f955aa627995397c5f6d0db9eb5965e83950079884886abef382fd679b"}
Jul 24 18:08:24.208536 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.208403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"
Jul 24 18:08:24.209629 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.209602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2" event={"ID":"4d6c65d7-75b9-4c97-806c-9c89e9d812bb","Type":"ContainerStarted","Data":"daf8e9744af9d9885c25660089e5fe0fa2fcbba8fcd65956df3136f6d989ed43"}
Jul 24 18:08:24.225627 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.225526    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g" podStartSLOduration=1.225509738 podStartE2EDuration="1.225509738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:08:24.224314583 +0000 UTC m=+1917.740825617" watchObservedRunningTime="2026-07-24 18:08:24.225509738 +0000 UTC m=+1917.742020773"
Jul 24 18:08:24.529522 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.529498    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:08:24.589078 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589039    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-credential-socket\") pod \"a491614f-c757-466d-9ccc-ce849610c87b\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") "
Jul 24 18:08:24.589078 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589079    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-workload-socket\") pod \"a491614f-c757-466d-9ccc-ce849610c87b\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") "
Jul 24 18:08:24.589307 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589095    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a491614f-c757-466d-9ccc-ce849610c87b-istiod-ca-cert\") pod \"a491614f-c757-466d-9ccc-ce849610c87b\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") "
Jul 24 18:08:24.589307 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcqhh\" (UniqueName: \"kubernetes.io/projected/a491614f-c757-466d-9ccc-ce849610c87b-kube-api-access-mcqhh\") pod \"a491614f-c757-466d-9ccc-ce849610c87b\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") "
Jul 24 18:08:24.589307 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589190    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-workload-certs\") pod \"a491614f-c757-466d-9ccc-ce849610c87b\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") "
Jul 24 18:08:24.589307 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589215    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a491614f-c757-466d-9ccc-ce849610c87b-istio-token\") pod \"a491614f-c757-466d-9ccc-ce849610c87b\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") "
Jul 24 18:08:24.589307 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589258    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-istio-data\") pod \"a491614f-c757-466d-9ccc-ce849610c87b\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") "
Jul 24 18:08:24.589307 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589304    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-istio-envoy\") pod \"a491614f-c757-466d-9ccc-ce849610c87b\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") "
Jul 24 18:08:24.589621 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589341    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a491614f-c757-466d-9ccc-ce849610c87b-istio-podinfo\") pod \"a491614f-c757-466d-9ccc-ce849610c87b\" (UID: \"a491614f-c757-466d-9ccc-ce849610c87b\") "
Jul 24 18:08:24.589686 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589335    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a491614f-c757-466d-9ccc-ce849610c87b" (UID: "a491614f-c757-466d-9ccc-ce849610c87b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:08:24.589686 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a491614f-c757-466d-9ccc-ce849610c87b" (UID: "a491614f-c757-466d-9ccc-ce849610c87b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:08:24.589686 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589549    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a491614f-c757-466d-9ccc-ce849610c87b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a491614f-c757-466d-9ccc-ce849610c87b" (UID: "a491614f-c757-466d-9ccc-ce849610c87b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:08:24.589861 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.589824    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a491614f-c757-466d-9ccc-ce849610c87b" (UID: "a491614f-c757-466d-9ccc-ce849610c87b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:08:24.590052 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.590031    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-istio-data" (OuterVolumeSpecName: "istio-data") pod "a491614f-c757-466d-9ccc-ce849610c87b" (UID: "a491614f-c757-466d-9ccc-ce849610c87b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:08:24.591844 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.591814    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a491614f-c757-466d-9ccc-ce849610c87b" (UID: "a491614f-c757-466d-9ccc-ce849610c87b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:08:24.591844 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.591829    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a491614f-c757-466d-9ccc-ce849610c87b-kube-api-access-mcqhh" (OuterVolumeSpecName: "kube-api-access-mcqhh") pod "a491614f-c757-466d-9ccc-ce849610c87b" (UID: "a491614f-c757-466d-9ccc-ce849610c87b"). InnerVolumeSpecName "kube-api-access-mcqhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:08:24.591998 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.591864    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a491614f-c757-466d-9ccc-ce849610c87b-istio-token" (OuterVolumeSpecName: "istio-token") pod "a491614f-c757-466d-9ccc-ce849610c87b" (UID: "a491614f-c757-466d-9ccc-ce849610c87b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:08:24.591998 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.591938    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a491614f-c757-466d-9ccc-ce849610c87b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a491614f-c757-466d-9ccc-ce849610c87b" (UID: "a491614f-c757-466d-9ccc-ce849610c87b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 18:08:24.690876 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.690844    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:24.690876 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.690875    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:24.691020 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.690891    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a491614f-c757-466d-9ccc-ce849610c87b-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:24.691020 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.690904    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:24.691020 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.690917    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:24.691020 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.690930    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a491614f-c757-466d-9ccc-ce849610c87b-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:24.691020 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.690943    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcqhh\" (UniqueName: \"kubernetes.io/projected/a491614f-c757-466d-9ccc-ce849610c87b-kube-api-access-mcqhh\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:24.691020 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.690956    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a491614f-c757-466d-9ccc-ce849610c87b-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:24.691020 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:24.690968    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a491614f-c757-466d-9ccc-ce849610c87b-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:25.216012 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.215927    2581 generic.go:356] "Generic (PLEG): container finished" podID="a491614f-c757-466d-9ccc-ce849610c87b" containerID="b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175" exitCode=0
Jul 24 18:08:25.216012 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.215990    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"
Jul 24 18:08:25.216481 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.216015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz" event={"ID":"a491614f-c757-466d-9ccc-ce849610c87b","Type":"ContainerDied","Data":"b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175"}
Jul 24 18:08:25.216481 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.216076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz" event={"ID":"a491614f-c757-466d-9ccc-ce849610c87b","Type":"ContainerDied","Data":"c4a5755d4f93b2e382db42c5f1b0dc615c5adc208a3d12940c198f4e6740f37e"}
Jul 24 18:08:25.216481 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.216104    2581 scope.go:117] "RemoveContainer" containerID="b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175"
Jul 24 18:08:25.218116 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.217804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2" event={"ID":"4d6c65d7-75b9-4c97-806c-9c89e9d812bb","Type":"ContainerStarted","Data":"dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0"}
Jul 24 18:08:25.218116 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.217956    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:25.224407 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.224330    2581 scope.go:117] "RemoveContainer" containerID="006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c"
Jul 24 18:08:25.232266 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.232238    2581 scope.go:117] "RemoveContainer" containerID="b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175"
Jul 24 18:08:25.232521 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:08:25.232501    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175\": container with ID starting with b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175 not found: ID does not exist" containerID="b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175"
Jul 24 18:08:25.232609 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.232531    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175"} err="failed to get container status \"b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175\": rpc error: code = NotFound desc = could not find container \"b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175\": container with ID starting with b53268ec4f984cb6f844f694154e53b7ad31852e65932479e7502c12172d0175 not found: ID does not exist"
Jul 24 18:08:25.232609 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.232547    2581 scope.go:117] "RemoveContainer" containerID="006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c"
Jul 24 18:08:25.232800 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:08:25.232782    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c\": container with ID starting with 006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c not found: ID does not exist" containerID="006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c"
Jul 24 18:08:25.232853 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.232809    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c"} err="failed to get container status \"006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c\": rpc error: code = NotFound desc = could not find container \"006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c\": container with ID starting with 006f7c7405ed45a84e08b7015d1a133827da4b793ae776e646bb86610afc516c not found: ID does not exist"
Jul 24 18:08:25.233411 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.233394    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"]
Jul 24 18:08:25.237260 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.237239    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d49fe598-openshift-default-6b96db679b-x4zfz"]
Jul 24 18:08:25.252124 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:25.252073    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2" podStartSLOduration=2.79463654 podStartE2EDuration="3.2520499s" totalImagesPullingTime="457.41336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:08:25.251223182 +0000 UTC m=+1918.767734220" watchObservedRunningTime="2026-07-24 18:08:25.2520499 +0000 UTC m=+1918.768560935"
Jul 24 18:08:26.957940 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:26.957900    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a491614f-c757-466d-9ccc-ce849610c87b" path="/var/lib/kubelet/pods/a491614f-c757-466d-9ccc-ce849610c87b/volumes"
Jul 24 18:08:31.230923 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:31.230894    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:35.220275 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:35.220232    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g" podUID="674a0404-56a2-49b3-967e-706f12b5cff3" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.28:9004: connect: connection refused"
Jul 24 18:08:36.277147 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:36.277115    2581 generic.go:356] "Generic (PLEG): container finished" podID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerID="f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6" exitCode=0
Jul 24 18:08:36.277591 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:36.277184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz" event={"ID":"70b84616-1fec-4e79-a32a-ff4ee81b994d","Type":"ContainerDied","Data":"f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6"}
Jul 24 18:08:36.277697 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:36.277680    2581 scope.go:117] "RemoveContainer" containerID="f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6"
Jul 24 18:08:36.278677 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:36.278647    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:08:36.278769 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:36.278753    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:08:36.278809 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:36.278794    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:08:37.200069 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:37.200028    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:37.283363 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:37.283326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz" event={"ID":"70b84616-1fec-4e79-a32a-ff4ee81b994d","Type":"ContainerStarted","Data":"207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481"}
Jul 24 18:08:37.640320 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:37.640287    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:37.641719 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:37.641681    2581 patch_prober.go:28] interesting pod/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Jul 24 18:08:37.641963 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:37.641935    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz" podUID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Jul 24 18:08:38.640357 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:38.640324    2581 patch_prober.go:28] interesting pod/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Jul 24 18:08:38.640788 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:38.640379    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz" podUID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Jul 24 18:08:39.639858 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:39.639818    2581 patch_prober.go:28] interesting pod/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Jul 24 18:08:39.640028 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:39.639893    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz" podUID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Jul 24 18:08:40.639910 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:40.639874    2581 patch_prober.go:28] interesting pod/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Jul 24 18:08:40.640295 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:40.639929    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz" podUID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Jul 24 18:08:41.643961 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:41.643929    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:41.644383 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:41.644218    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:41.644899 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:41.644884    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:08:45.218463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:45.218434    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"
Jul 24 18:08:47.740400 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:47.740367    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh"]
Jul 24 18:08:47.740821 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:47.740711    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a491614f-c757-466d-9ccc-ce849610c87b" containerName="istio-proxy"
Jul 24 18:08:47.740821 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:47.740726    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a491614f-c757-466d-9ccc-ce849610c87b" containerName="istio-proxy"
Jul 24 18:08:47.740821 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:47.740741    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a491614f-c757-466d-9ccc-ce849610c87b" containerName="istio-proxy"
Jul 24 18:08:47.740821 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:47.740747    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a491614f-c757-466d-9ccc-ce849610c87b" containerName="istio-proxy"
Jul 24 18:08:47.740821 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:47.740800    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a491614f-c757-466d-9ccc-ce849610c87b" containerName="istio-proxy"
Jul 24 18:08:47.743942 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:47.743923    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh"
Jul 24 18:08:47.746378 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:47.746360    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v7cm8\""
Jul 24 18:08:47.750738 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:47.750626    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh"]
Jul 24 18:08:47.890615 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:47.890560    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b96747-08d3-4e30-8c0f-d5aa46b0c0fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh\" (UID: \"84b96747-08d3-4e30-8c0f-d5aa46b0c0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh"
Jul 24 18:08:47.991261 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:47.991169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b96747-08d3-4e30-8c0f-d5aa46b0c0fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh\" (UID: \"84b96747-08d3-4e30-8c0f-d5aa46b0c0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh"
Jul 24 18:08:47.991810 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:47.991792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b96747-08d3-4e30-8c0f-d5aa46b0c0fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh\" (UID: \"84b96747-08d3-4e30-8c0f-d5aa46b0c0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh"
Jul 24 18:08:48.057472 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:48.057442    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh"
Jul 24 18:08:48.185097 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:48.185066    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh"]
Jul 24 18:08:48.188877 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:08:48.188849    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b96747_08d3_4e30_8c0f_d5aa46b0c0fc.slice/crio-27d1dd71a6180dfbbd90f4034de1cb06c106c801fd7b6baeaf8368847519b778 WatchSource:0}: Error finding container 27d1dd71a6180dfbbd90f4034de1cb06c106c801fd7b6baeaf8368847519b778: Status 404 returned error can't find the container with id 27d1dd71a6180dfbbd90f4034de1cb06c106c801fd7b6baeaf8368847519b778
Jul 24 18:08:48.328642 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:48.328606    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh" event={"ID":"84b96747-08d3-4e30-8c0f-d5aa46b0c0fc","Type":"ContainerStarted","Data":"27d1dd71a6180dfbbd90f4034de1cb06c106c801fd7b6baeaf8368847519b778"}
Jul 24 18:08:51.344842 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:51.344803    2581 generic.go:356] "Generic (PLEG): container finished" podID="84b96747-08d3-4e30-8c0f-d5aa46b0c0fc" containerID="136c1b6ce76d26b0753aa9b58345719b30e874c07f5fc900f3bee173dd383e3b" exitCode=0
Jul 24 18:08:51.345198 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:51.344847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh" event={"ID":"84b96747-08d3-4e30-8c0f-d5aa46b0c0fc","Type":"ContainerDied","Data":"136c1b6ce76d26b0753aa9b58345719b30e874c07f5fc900f3bee173dd383e3b"}
Jul 24 18:08:52.475039 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:52.475016    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh"
Jul 24 18:08:52.634394 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:52.634283    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b96747-08d3-4e30-8c0f-d5aa46b0c0fc-maas-api-service-ca\") pod \"84b96747-08d3-4e30-8c0f-d5aa46b0c0fc\" (UID: \"84b96747-08d3-4e30-8c0f-d5aa46b0c0fc\") "
Jul 24 18:08:52.634658 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:52.634636    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b96747-08d3-4e30-8c0f-d5aa46b0c0fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "84b96747-08d3-4e30-8c0f-d5aa46b0c0fc" (UID: "84b96747-08d3-4e30-8c0f-d5aa46b0c0fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:08:52.735329 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:52.735296    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b96747-08d3-4e30-8c0f-d5aa46b0c0fc-maas-api-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:53.354008 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:53.353974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh" event={"ID":"84b96747-08d3-4e30-8c0f-d5aa46b0c0fc","Type":"ContainerDied","Data":"27d1dd71a6180dfbbd90f4034de1cb06c106c801fd7b6baeaf8368847519b778"}
Jul 24 18:08:53.354008 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:53.354003    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh"
Jul 24 18:08:53.354218 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:53.354021    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27d1dd71a6180dfbbd90f4034de1cb06c106c801fd7b6baeaf8368847519b778"
Jul 24 18:08:58.356420 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.356386    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"]
Jul 24 18:08:58.356901 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.356649    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2" podUID="4d6c65d7-75b9-4c97-806c-9c89e9d812bb" containerName="maas-api" containerID="cri-o://dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0" gracePeriod=30
Jul 24 18:08:58.385834 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.385803    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:58.425757 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.425729    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"]
Jul 24 18:08:58.425953 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.425931    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g" podUID="674a0404-56a2-49b3-967e-706f12b5cff3" containerName="payload-pre-processing" containerID="cri-o://20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c" gracePeriod=30
Jul 24 18:08:58.460769 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.460740    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:58.562463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.562431    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:08:58.688237 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.688214    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:58.690675 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.690656    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"
Jul 24 18:08:58.784406 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.784373    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xqjj\" (UniqueName: \"kubernetes.io/projected/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-kube-api-access-2xqjj\") pod \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") "
Jul 24 18:08:58.784596 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.784425    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cchc\" (UniqueName: \"kubernetes.io/projected/674a0404-56a2-49b3-967e-706f12b5cff3-kube-api-access-2cchc\") pod \"674a0404-56a2-49b3-967e-706f12b5cff3\" (UID: \"674a0404-56a2-49b3-967e-706f12b5cff3\") "
Jul 24 18:08:58.784596 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.784457    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-maas-api-tls\") pod \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") "
Jul 24 18:08:58.784596 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.784550    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-openshift-service-ca\") pod \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\" (UID: \"4d6c65d7-75b9-4c97-806c-9c89e9d812bb\") "
Jul 24 18:08:58.784781 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.784602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/674a0404-56a2-49b3-967e-706f12b5cff3-plugins-config-volume\") pod \"674a0404-56a2-49b3-967e-706f12b5cff3\" (UID: \"674a0404-56a2-49b3-967e-706f12b5cff3\") "
Jul 24 18:08:58.784973 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.784941    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d6c65d7-75b9-4c97-806c-9c89e9d812bb" (UID: "4d6c65d7-75b9-4c97-806c-9c89e9d812bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:08:58.785038 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.785005    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674a0404-56a2-49b3-967e-706f12b5cff3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "674a0404-56a2-49b3-967e-706f12b5cff3" (UID: "674a0404-56a2-49b3-967e-706f12b5cff3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:08:58.786612 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.786589    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-kube-api-access-2xqjj" (OuterVolumeSpecName: "kube-api-access-2xqjj") pod "4d6c65d7-75b9-4c97-806c-9c89e9d812bb" (UID: "4d6c65d7-75b9-4c97-806c-9c89e9d812bb"). InnerVolumeSpecName "kube-api-access-2xqjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:08:58.786943 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.786928    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674a0404-56a2-49b3-967e-706f12b5cff3-kube-api-access-2cchc" (OuterVolumeSpecName: "kube-api-access-2cchc") pod "674a0404-56a2-49b3-967e-706f12b5cff3" (UID: "674a0404-56a2-49b3-967e-706f12b5cff3"). InnerVolumeSpecName "kube-api-access-2cchc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:08:58.786986 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.786951    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d6c65d7-75b9-4c97-806c-9c89e9d812bb" (UID: "4d6c65d7-75b9-4c97-806c-9c89e9d812bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:08:58.886220 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.886134    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xqjj\" (UniqueName: \"kubernetes.io/projected/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-kube-api-access-2xqjj\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:58.886220 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.886166    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cchc\" (UniqueName: \"kubernetes.io/projected/674a0404-56a2-49b3-967e-706f12b5cff3-kube-api-access-2cchc\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:58.886220 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.886178    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:58.886220 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.886191    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c65d7-75b9-4c97-806c-9c89e9d812bb-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:58.886220 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:58.886202    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/674a0404-56a2-49b3-967e-706f12b5cff3-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:08:59.380565 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.380531    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d6c65d7-75b9-4c97-806c-9c89e9d812bb" containerID="dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0" exitCode=0
Jul 24 18:08:59.381023 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.380594    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"
Jul 24 18:08:59.381023 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.380610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2" event={"ID":"4d6c65d7-75b9-4c97-806c-9c89e9d812bb","Type":"ContainerDied","Data":"dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0"}
Jul 24 18:08:59.381023 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.380664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2" event={"ID":"4d6c65d7-75b9-4c97-806c-9c89e9d812bb","Type":"ContainerDied","Data":"daf8e9744af9d9885c25660089e5fe0fa2fcbba8fcd65956df3136f6d989ed43"}
Jul 24 18:08:59.381023 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.380686    2581 scope.go:117] "RemoveContainer" containerID="dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0"
Jul 24 18:08:59.382068 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.382047    2581 generic.go:356] "Generic (PLEG): container finished" podID="674a0404-56a2-49b3-967e-706f12b5cff3" containerID="20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c" exitCode=0
Jul 24 18:08:59.382149 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.382079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g" event={"ID":"674a0404-56a2-49b3-967e-706f12b5cff3","Type":"ContainerDied","Data":"20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c"}
Jul 24 18:08:59.382149 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.382091    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"
Jul 24 18:08:59.382149 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.382109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g" event={"ID":"674a0404-56a2-49b3-967e-706f12b5cff3","Type":"ContainerDied","Data":"9be952f955aa627995397c5f6d0db9eb5965e83950079884886abef382fd679b"}
Jul 24 18:08:59.388832 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.388815    2581 scope.go:117] "RemoveContainer" containerID="dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0"
Jul 24 18:08:59.389623 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:08:59.389561    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0\": container with ID starting with dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0 not found: ID does not exist" containerID="dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0"
Jul 24 18:08:59.389766 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.389740    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0"} err="failed to get container status \"dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0\": rpc error: code = NotFound desc = could not find container \"dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0\": container with ID starting with dea62769bca112e67780831b8f5141e52cb336a5cddfebd499b16977cb5878f0 not found: ID does not exist"
Jul 24 18:08:59.389868 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.389856    2581 scope.go:117] "RemoveContainer" containerID="20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c"
Jul 24 18:08:59.401495 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.401471    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"]
Jul 24 18:08:59.402082 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.402066    2581 scope.go:117] "RemoveContainer" containerID="20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c"
Jul 24 18:08:59.402332 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:08:59.402313    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c\": container with ID starting with 20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c not found: ID does not exist" containerID="20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c"
Jul 24 18:08:59.402396 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.402339    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c"} err="failed to get container status \"20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c\": rpc error: code = NotFound desc = could not find container \"20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c\": container with ID starting with 20a9ccec8390b28a795e6d3bed69980a2949e887722d210e049ddf54f3c2608c not found: ID does not exist"
Jul 24 18:08:59.406643 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.406623    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-23b920dc-6699cf9fc6-bcxz2"]
Jul 24 18:08:59.420588 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.420547    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"]
Jul 24 18:08:59.428055 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:08:59.428036    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-23b920dc-7ddfb5ff9b-bmq7g"]
Jul 24 18:09:00.958391 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:00.958361    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d6c65d7-75b9-4c97-806c-9c89e9d812bb" path="/var/lib/kubelet/pods/4d6c65d7-75b9-4c97-806c-9c89e9d812bb/volumes"
Jul 24 18:09:00.958763 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:00.958714    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="674a0404-56a2-49b3-967e-706f12b5cff3" path="/var/lib/kubelet/pods/674a0404-56a2-49b3-967e-706f12b5cff3/volumes"
Jul 24 18:09:02.827194 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:02.827160    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:02.828428 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:02.828408    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh"]
Jul 24 18:09:02.830712 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:02.830691    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-23b920dc-zvsgh"]
Jul 24 18:09:02.959847 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:02.959816    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84b96747-08d3-4e30-8c0f-d5aa46b0c0fc" path="/var/lib/kubelet/pods/84b96747-08d3-4e30-8c0f-d5aa46b0c0fc/volumes"
Jul 24 18:09:02.966293 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:02.966268    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"]
Jul 24 18:09:02.966608 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:02.966555    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz" podUID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerName="istio-proxy" containerID="cri-o://207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481" gracePeriod=30
Jul 24 18:09:02.996931 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:02.996904    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:03.245744 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.245662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:03.484300 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.484265    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:03.775249 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.775216    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"]
Jul 24 18:09:03.775506 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.775494    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d6c65d7-75b9-4c97-806c-9c89e9d812bb" containerName="maas-api"
Jul 24 18:09:03.775546 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.775508    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d6c65d7-75b9-4c97-806c-9c89e9d812bb" containerName="maas-api"
Jul 24 18:09:03.775546 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.775526    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674a0404-56a2-49b3-967e-706f12b5cff3" containerName="payload-pre-processing"
Jul 24 18:09:03.775546 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.775532    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="674a0404-56a2-49b3-967e-706f12b5cff3" containerName="payload-pre-processing"
Jul 24 18:09:03.775546 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.775540    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b96747-08d3-4e30-8c0f-d5aa46b0c0fc" containerName="revoke-keys"
Jul 24 18:09:03.775546 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.775547    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b96747-08d3-4e30-8c0f-d5aa46b0c0fc" containerName="revoke-keys"
Jul 24 18:09:03.775729 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.775611    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="674a0404-56a2-49b3-967e-706f12b5cff3" containerName="payload-pre-processing"
Jul 24 18:09:03.775729 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.775622    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a491614f-c757-466d-9ccc-ce849610c87b" containerName="istio-proxy"
Jul 24 18:09:03.775729 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.775628    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d6c65d7-75b9-4c97-806c-9c89e9d812bb" containerName="maas-api"
Jul 24 18:09:03.775729 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.775634    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b96747-08d3-4e30-8c0f-d5aa46b0c0fc" containerName="revoke-keys"
Jul 24 18:09:03.779620 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.779597    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:03.782682 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.782656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b716847d-openshift-default-dockercfg-7h2d5\""
Jul 24 18:09:03.792395 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.792373    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"]
Jul 24 18:09:03.808489 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.808466    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:03.927298 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.927263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-istio-data\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:03.927298 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.927300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-workload-socket\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:03.927713 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.927324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-workload-certs\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:03.927713 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.927438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-istio-envoy\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:03.927713 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.927467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da464976-b8ce-4770-a7a7-83d645f42526-istiod-ca-cert\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:03.927713 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.927500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da464976-b8ce-4770-a7a7-83d645f42526-istio-podinfo\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:03.927713 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.927525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da464976-b8ce-4770-a7a7-83d645f42526-istio-token\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:03.927713 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.927580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99nd9\" (UniqueName: \"kubernetes.io/projected/da464976-b8ce-4770-a7a7-83d645f42526-kube-api-access-99nd9\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:03.927713 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:03.927615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-credential-socket\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.021096 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.021058    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:04.028214 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-istio-data\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.028214 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028185    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-workload-socket\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.028214 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-workload-certs\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.028453 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-istio-envoy\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.028453 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da464976-b8ce-4770-a7a7-83d645f42526-istiod-ca-cert\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.028453 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da464976-b8ce-4770-a7a7-83d645f42526-istio-podinfo\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.028453 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da464976-b8ce-4770-a7a7-83d645f42526-istio-token\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.028453 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99nd9\" (UniqueName: \"kubernetes.io/projected/da464976-b8ce-4770-a7a7-83d645f42526-kube-api-access-99nd9\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.028453 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-credential-socket\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.028740 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-workload-socket\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.028740 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-workload-certs\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.028740 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-istio-data\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.028834 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.028778    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-credential-socket\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.029182 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.029161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da464976-b8ce-4770-a7a7-83d645f42526-istiod-ca-cert\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.030855 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.030825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-istio-envoy\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.030932 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.030899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da464976-b8ce-4770-a7a7-83d645f42526-istio-podinfo\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.045910 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.045891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da464976-b8ce-4770-a7a7-83d645f42526-istio-token\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.046102 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.046084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99nd9\" (UniqueName: \"kubernetes.io/projected/da464976-b8ce-4770-a7a7-83d645f42526-kube-api-access-99nd9\") pod \"e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") " pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.094933 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.094906    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:04.229058 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.229027    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"]
Jul 24 18:09:04.232208 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:09:04.232181    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda464976_b8ce_4770_a7a7_83d645f42526.slice/crio-f2cae3d871f508daf3d0367fa0c434fa9d86bb98d6f0cdcebea272549aed59e7 WatchSource:0}: Error finding container f2cae3d871f508daf3d0367fa0c434fa9d86bb98d6f0cdcebea272549aed59e7: Status 404 returned error can't find the container with id f2cae3d871f508daf3d0367fa0c434fa9d86bb98d6f0cdcebea272549aed59e7
Jul 24 18:09:04.234309 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.234275    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:09:04.234375 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.234340    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:09:04.234375 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.234369    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:09:04.406279 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.406238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" event={"ID":"da464976-b8ce-4770-a7a7-83d645f42526","Type":"ContainerStarted","Data":"0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2"}
Jul 24 18:09:04.406445 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.406301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" event={"ID":"da464976-b8ce-4770-a7a7-83d645f42526","Type":"ContainerStarted","Data":"f2cae3d871f508daf3d0367fa0c434fa9d86bb98d6f0cdcebea272549aed59e7"}
Jul 24 18:09:04.428750 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:04.428694    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" podStartSLOduration=1.428678751 podStartE2EDuration="1.428678751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:09:04.426776452 +0000 UTC m=+1957.943287489" watchObservedRunningTime="2026-07-24 18:09:04.428678751 +0000 UTC m=+1957.945189852"
Jul 24 18:09:05.095728 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:05.095696    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:05.100663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:05.100640    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:05.411491 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:05.411413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:05.412219 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:05.412203    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:05.940160 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:05.940124    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:06.084957 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.084928    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"]
Jul 24 18:09:06.088257 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.088237    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:06.092535 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.092511    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b716847d-serving-cert\""
Jul 24 18:09:06.092675 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.092535    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fvm85\""
Jul 24 18:09:06.093204 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.093189    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:09:06.093440 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.093425    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:09:06.106779 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.106756    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"]
Jul 24 18:09:06.110909 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.110878    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:06.229455 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.229357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:06.250402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.250373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adde3481-ecda-4093-a808-ed32d7d2f668-openshift-service-ca\") pod \"maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt\" (UID: \"adde3481-ecda-4093-a808-ed32d7d2f668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:06.250530 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.250432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvpx6\" (UniqueName: \"kubernetes.io/projected/adde3481-ecda-4093-a808-ed32d7d2f668-kube-api-access-gvpx6\") pod \"maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt\" (UID: \"adde3481-ecda-4093-a808-ed32d7d2f668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:06.250530 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.250489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adde3481-ecda-4093-a808-ed32d7d2f668-maas-api-tls\") pod \"maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt\" (UID: \"adde3481-ecda-4093-a808-ed32d7d2f668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:06.351782 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.351748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adde3481-ecda-4093-a808-ed32d7d2f668-openshift-service-ca\") pod \"maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt\" (UID: \"adde3481-ecda-4093-a808-ed32d7d2f668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:06.351915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.351810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvpx6\" (UniqueName: \"kubernetes.io/projected/adde3481-ecda-4093-a808-ed32d7d2f668-kube-api-access-gvpx6\") pod \"maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt\" (UID: \"adde3481-ecda-4093-a808-ed32d7d2f668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:06.351915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.351831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adde3481-ecda-4093-a808-ed32d7d2f668-maas-api-tls\") pod \"maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt\" (UID: \"adde3481-ecda-4093-a808-ed32d7d2f668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:06.352384 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.352363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adde3481-ecda-4093-a808-ed32d7d2f668-openshift-service-ca\") pod \"maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt\" (UID: \"adde3481-ecda-4093-a808-ed32d7d2f668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:06.354244 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.354225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adde3481-ecda-4093-a808-ed32d7d2f668-maas-api-tls\") pod \"maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt\" (UID: \"adde3481-ecda-4093-a808-ed32d7d2f668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:06.354611 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.354591    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:06.362250 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.362233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvpx6\" (UniqueName: \"kubernetes.io/projected/adde3481-ecda-4093-a808-ed32d7d2f668-kube-api-access-gvpx6\") pod \"maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt\" (UID: \"adde3481-ecda-4093-a808-ed32d7d2f668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:06.399117 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.399096    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:06.529937 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.529906    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"]
Jul 24 18:09:06.531524 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:09:06.531498    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadde3481_ecda_4093_a808_ed32d7d2f668.slice/crio-5bca00aec6f1b73d89f851380877bf8be502c88b13315708a6ffe791b6b7e5f1 WatchSource:0}: Error finding container 5bca00aec6f1b73d89f851380877bf8be502c88b13315708a6ffe791b6b7e5f1: Status 404 returned error can't find the container with id 5bca00aec6f1b73d89f851380877bf8be502c88b13315708a6ffe791b6b7e5f1
Jul 24 18:09:06.731149 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.731120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:06.825115 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.825087    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:06.871413 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.871384    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"]
Jul 24 18:09:06.875800 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.875782    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"
Jul 24 18:09:06.878606 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.878587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b716847d-dockercfg-lknql\""
Jul 24 18:09:06.879405 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.879387    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b716847d\""
Jul 24 18:09:06.891462 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:06.891439    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"]
Jul 24 18:09:07.058300 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.058269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83-plugins-config-volume\") pod \"payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf\" (UID: \"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83\") " pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"
Jul 24 18:09:07.058465 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.058331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dxlk\" (UniqueName: \"kubernetes.io/projected/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83-kube-api-access-4dxlk\") pod \"payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf\" (UID: \"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83\") " pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"
Jul 24 18:09:07.159748 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.159668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dxlk\" (UniqueName: \"kubernetes.io/projected/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83-kube-api-access-4dxlk\") pod \"payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf\" (UID: \"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83\") " pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"
Jul 24 18:09:07.160184 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.159761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83-plugins-config-volume\") pod \"payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf\" (UID: \"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83\") " pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"
Jul 24 18:09:07.160411 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.160392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83-plugins-config-volume\") pod \"payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf\" (UID: \"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83\") " pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"
Jul 24 18:09:07.168435 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.168409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dxlk\" (UniqueName: \"kubernetes.io/projected/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83-kube-api-access-4dxlk\") pod \"payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf\" (UID: \"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83\") " pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"
Jul 24 18:09:07.186222 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.186200    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"
Jul 24 18:09:07.322911 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.322890    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"]
Jul 24 18:09:07.325414 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:09:07.325385    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf8c4ce_5dfe_4847_a87b_ca1ff468ef83.slice/crio-30e4153990754979704dcdecc1cf327a3df8a09b0555c68c33e2bd53f7822fb0 WatchSource:0}: Error finding container 30e4153990754979704dcdecc1cf327a3df8a09b0555c68c33e2bd53f7822fb0: Status 404 returned error can't find the container with id 30e4153990754979704dcdecc1cf327a3df8a09b0555c68c33e2bd53f7822fb0
Jul 24 18:09:07.423093 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.423021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf" event={"ID":"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83","Type":"ContainerStarted","Data":"dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20"}
Jul 24 18:09:07.423093 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.423080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf" event={"ID":"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83","Type":"ContainerStarted","Data":"30e4153990754979704dcdecc1cf327a3df8a09b0555c68c33e2bd53f7822fb0"}
Jul 24 18:09:07.423304 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.423113    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"
Jul 24 18:09:07.424777 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.424750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt" event={"ID":"adde3481-ecda-4093-a808-ed32d7d2f668","Type":"ContainerStarted","Data":"7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e"}
Jul 24 18:09:07.424874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.424802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt" event={"ID":"adde3481-ecda-4093-a808-ed32d7d2f668","Type":"ContainerStarted","Data":"5bca00aec6f1b73d89f851380877bf8be502c88b13315708a6ffe791b6b7e5f1"}
Jul 24 18:09:07.424968 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.424940    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:07.440366 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.440324    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf" podStartSLOduration=1.440312144 podStartE2EDuration="1.440312144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:09:07.439227335 +0000 UTC m=+1960.955738373" watchObservedRunningTime="2026-07-24 18:09:07.440312144 +0000 UTC m=+1960.956823182"
Jul 24 18:09:07.459466 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:07.459415    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt" podStartSLOduration=1.08052242 podStartE2EDuration="1.459398204s" totalImagesPullingTime="378.875784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:09:07.458174205 +0000 UTC m=+1960.974685241" watchObservedRunningTime="2026-07-24 18:09:07.459398204 +0000 UTC m=+1960.975909241"
Jul 24 18:09:08.207067 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.207046    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:09:08.268410 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.268344    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-data\") pod \"70b84616-1fec-4e79-a32a-ff4ee81b994d\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") "
Jul 24 18:09:08.268410 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.268380    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-token\") pod \"70b84616-1fec-4e79-a32a-ff4ee81b994d\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") "
Jul 24 18:09:08.268610 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.268414    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-credential-socket\") pod \"70b84616-1fec-4e79-a32a-ff4ee81b994d\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") "
Jul 24 18:09:08.268610 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.268439    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-workload-socket\") pod \"70b84616-1fec-4e79-a32a-ff4ee81b994d\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") "
Jul 24 18:09:08.268610 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.268471    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-podinfo\") pod \"70b84616-1fec-4e79-a32a-ff4ee81b994d\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") "
Jul 24 18:09:08.268610 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.268514    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-workload-certs\") pod \"70b84616-1fec-4e79-a32a-ff4ee81b994d\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") "
Jul 24 18:09:08.268610 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.268541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4slt\" (UniqueName: \"kubernetes.io/projected/70b84616-1fec-4e79-a32a-ff4ee81b994d-kube-api-access-s4slt\") pod \"70b84616-1fec-4e79-a32a-ff4ee81b994d\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") "
Jul 24 18:09:08.268610 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.268588    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b84616-1fec-4e79-a32a-ff4ee81b994d-istiod-ca-cert\") pod \"70b84616-1fec-4e79-a32a-ff4ee81b994d\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") "
Jul 24 18:09:08.268917 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.268615    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-envoy\") pod \"70b84616-1fec-4e79-a32a-ff4ee81b994d\" (UID: \"70b84616-1fec-4e79-a32a-ff4ee81b994d\") "
Jul 24 18:09:08.268917 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.268805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "70b84616-1fec-4e79-a32a-ff4ee81b994d" (UID: "70b84616-1fec-4e79-a32a-ff4ee81b994d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:09:08.268917 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.268816    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "70b84616-1fec-4e79-a32a-ff4ee81b994d" (UID: "70b84616-1fec-4e79-a32a-ff4ee81b994d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:09:08.269080 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.269016    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "70b84616-1fec-4e79-a32a-ff4ee81b994d" (UID: "70b84616-1fec-4e79-a32a-ff4ee81b994d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:09:08.269137 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.269077    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b84616-1fec-4e79-a32a-ff4ee81b994d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "70b84616-1fec-4e79-a32a-ff4ee81b994d" (UID: "70b84616-1fec-4e79-a32a-ff4ee81b994d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:09:08.269250 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.269225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-data" (OuterVolumeSpecName: "istio-data") pod "70b84616-1fec-4e79-a32a-ff4ee81b994d" (UID: "70b84616-1fec-4e79-a32a-ff4ee81b994d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:09:08.270828 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.270805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-token" (OuterVolumeSpecName: "istio-token") pod "70b84616-1fec-4e79-a32a-ff4ee81b994d" (UID: "70b84616-1fec-4e79-a32a-ff4ee81b994d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:09:08.270942 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.270881    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "70b84616-1fec-4e79-a32a-ff4ee81b994d" (UID: "70b84616-1fec-4e79-a32a-ff4ee81b994d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 18:09:08.271181 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.271159    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "70b84616-1fec-4e79-a32a-ff4ee81b994d" (UID: "70b84616-1fec-4e79-a32a-ff4ee81b994d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:09:08.271253 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.271167    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b84616-1fec-4e79-a32a-ff4ee81b994d-kube-api-access-s4slt" (OuterVolumeSpecName: "kube-api-access-s4slt") pod "70b84616-1fec-4e79-a32a-ff4ee81b994d" (UID: "70b84616-1fec-4e79-a32a-ff4ee81b994d"). InnerVolumeSpecName "kube-api-access-s4slt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:09:08.369706 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.369679    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:08.369706 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.369702    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:08.369706 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.369711    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4slt\" (UniqueName: \"kubernetes.io/projected/70b84616-1fec-4e79-a32a-ff4ee81b994d-kube-api-access-s4slt\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:08.369875 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.369722    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b84616-1fec-4e79-a32a-ff4ee81b994d-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:08.369875 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.369731    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:08.369875 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.369738    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:08.369875 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.369745    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b84616-1fec-4e79-a32a-ff4ee81b994d-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:08.369875 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.369753    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:08.369875 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.369760    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b84616-1fec-4e79-a32a-ff4ee81b994d-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:08.430157 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.430124    2581 generic.go:356] "Generic (PLEG): container finished" podID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerID="207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481" exitCode=0
Jul 24 18:09:08.430300 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.430181    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"
Jul 24 18:09:08.430300 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.430217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz" event={"ID":"70b84616-1fec-4e79-a32a-ff4ee81b994d","Type":"ContainerDied","Data":"207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481"}
Jul 24 18:09:08.430300 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.430282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz" event={"ID":"70b84616-1fec-4e79-a32a-ff4ee81b994d","Type":"ContainerDied","Data":"ba393ac801a0b861c40e83f04125ea93978057c4e6416c77db72692ff0cc06fd"}
Jul 24 18:09:08.430454 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.430314    2581 scope.go:117] "RemoveContainer" containerID="207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481"
Jul 24 18:09:08.439059 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.439044    2581 scope.go:117] "RemoveContainer" containerID="f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6"
Jul 24 18:09:08.446313 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.446296    2581 scope.go:117] "RemoveContainer" containerID="207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481"
Jul 24 18:09:08.446554 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:09:08.446536    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481\": container with ID starting with 207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481 not found: ID does not exist" containerID="207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481"
Jul 24 18:09:08.446619 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.446563    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481"} err="failed to get container status \"207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481\": rpc error: code = NotFound desc = could not find container \"207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481\": container with ID starting with 207fe11484bf90a09c4aa6f8c4f09ed8d42bf0f1eb68bb254b8a1d3bf171c481 not found: ID does not exist"
Jul 24 18:09:08.446619 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.446615    2581 scope.go:117] "RemoveContainer" containerID="f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6"
Jul 24 18:09:08.446846 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:09:08.446828    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6\": container with ID starting with f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6 not found: ID does not exist" containerID="f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6"
Jul 24 18:09:08.446890 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.446850    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6"} err="failed to get container status \"f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6\": rpc error: code = NotFound desc = could not find container \"f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6\": container with ID starting with f37c512f56d0125656653657c051e9d64a41ec6a7277670e07caeef24cb8b7a6 not found: ID does not exist"
Jul 24 18:09:08.450779 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.450756    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"]
Jul 24 18:09:08.454713 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.454694    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-23b920dc-gw-openshift-default-64fd646b8-8cmpz"]
Jul 24 18:09:08.957879 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:08.957848    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70b84616-1fec-4e79-a32a-ff4ee81b994d" path="/var/lib/kubelet/pods/70b84616-1fec-4e79-a32a-ff4ee81b994d/volumes"
Jul 24 18:09:13.436124 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:13.436095    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:18.432711 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:18.432682    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"
Jul 24 18:09:20.415033 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:20.414533    2581 patch_prober.go:28] interesting pod/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": read tcp 10.132.0.2:51128->10.132.0.30:15021: read: connection reset by peer" start-of-body=
Jul 24 18:09:20.415033 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:20.414659    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" podUID="da464976-b8ce-4770-a7a7-83d645f42526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": read tcp 10.132.0.2:51128->10.132.0.30:15021: read: connection reset by peer"
Jul 24 18:09:20.488824 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:20.488794    2581 generic.go:356] "Generic (PLEG): container finished" podID="da464976-b8ce-4770-a7a7-83d645f42526" containerID="0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2" exitCode=0
Jul 24 18:09:20.488950 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:20.488870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" event={"ID":"da464976-b8ce-4770-a7a7-83d645f42526","Type":"ContainerDied","Data":"0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2"}
Jul 24 18:09:20.489256 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:20.489235    2581 scope.go:117] "RemoveContainer" containerID="0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2"
Jul 24 18:09:20.490274 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:20.490241    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:09:20.490385 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:20.490312    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:09:20.490385 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:20.490349    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:09:21.495134 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:21.495095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" event={"ID":"da464976-b8ce-4770-a7a7-83d645f42526","Type":"ContainerStarted","Data":"b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9"}
Jul 24 18:09:22.095407 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:22.095374    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:22.095707 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:22.095683    2581 patch_prober.go:28] interesting pod/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body=
Jul 24 18:09:22.095760 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:22.095734    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" podUID="da464976-b8ce-4770-a7a7-83d645f42526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused"
Jul 24 18:09:23.095694 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:23.095656    2581 patch_prober.go:28] interesting pod/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body=
Jul 24 18:09:23.096165 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:23.095722    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" podUID="da464976-b8ce-4770-a7a7-83d645f42526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused"
Jul 24 18:09:24.096143 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:24.096102    2581 patch_prober.go:28] interesting pod/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body=
Jul 24 18:09:24.096535 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:24.096162    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" podUID="da464976-b8ce-4770-a7a7-83d645f42526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused"
Jul 24 18:09:25.096214 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:25.096172    2581 patch_prober.go:28] interesting pod/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body=
Jul 24 18:09:25.096635 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:25.096262    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" podUID="da464976-b8ce-4770-a7a7-83d645f42526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused"
Jul 24 18:09:26.099268 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:26.099237    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:26.099764 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:26.099629    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:26.100239 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:26.100219    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:42.177654 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.177619    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"]
Jul 24 18:09:42.178043 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.177873    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt" podUID="adde3481-ecda-4093-a808-ed32d7d2f668" containerName="maas-api" containerID="cri-o://7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e" gracePeriod=30
Jul 24 18:09:42.208792 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.208741    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:42.239453 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.239426    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"]
Jul 24 18:09:42.239655 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.239629    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf" podUID="baf8c4ce-5dfe-4847-a87b-ca1ff468ef83" containerName="payload-processing" containerID="cri-o://dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20" gracePeriod=30
Jul 24 18:09:42.504134 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.504112    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:42.536297 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.536269    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"
Jul 24 18:09:42.583467 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.583439    2581 generic.go:356] "Generic (PLEG): container finished" podID="baf8c4ce-5dfe-4847-a87b-ca1ff468ef83" containerID="dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20" exitCode=0
Jul 24 18:09:42.583640 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.583482    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"
Jul 24 18:09:42.583640 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.583507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf" event={"ID":"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83","Type":"ContainerDied","Data":"dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20"}
Jul 24 18:09:42.583640 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.583559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf" event={"ID":"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83","Type":"ContainerDied","Data":"30e4153990754979704dcdecc1cf327a3df8a09b0555c68c33e2bd53f7822fb0"}
Jul 24 18:09:42.583640 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.583621    2581 scope.go:117] "RemoveContainer" containerID="dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20"
Jul 24 18:09:42.584880 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.584857    2581 generic.go:356] "Generic (PLEG): container finished" podID="adde3481-ecda-4093-a808-ed32d7d2f668" containerID="7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e" exitCode=0
Jul 24 18:09:42.584992 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.584897    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"
Jul 24 18:09:42.584992 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.584912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt" event={"ID":"adde3481-ecda-4093-a808-ed32d7d2f668","Type":"ContainerDied","Data":"7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e"}
Jul 24 18:09:42.584992 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.584954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt" event={"ID":"adde3481-ecda-4093-a808-ed32d7d2f668","Type":"ContainerDied","Data":"5bca00aec6f1b73d89f851380877bf8be502c88b13315708a6ffe791b6b7e5f1"}
Jul 24 18:09:42.591345 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.591326    2581 scope.go:117] "RemoveContainer" containerID="dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20"
Jul 24 18:09:42.591603 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:09:42.591581    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20\": container with ID starting with dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20 not found: ID does not exist" containerID="dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20"
Jul 24 18:09:42.591703 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.591608    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20"} err="failed to get container status \"dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20\": rpc error: code = NotFound desc = could not find container \"dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20\": container with ID starting with dc3ed58622ba105aef2c85c88862bba59265c7034e518ea8c783820290c2ba20 not found: ID does not exist"
Jul 24 18:09:42.591703 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.591625    2581 scope.go:117] "RemoveContainer" containerID="7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e"
Jul 24 18:09:42.598564 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.598550    2581 scope.go:117] "RemoveContainer" containerID="7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e"
Jul 24 18:09:42.598900 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:09:42.598879    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e\": container with ID starting with 7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e not found: ID does not exist" containerID="7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e"
Jul 24 18:09:42.598963 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.598909    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e"} err="failed to get container status \"7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e\": rpc error: code = NotFound desc = could not find container \"7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e\": container with ID starting with 7bce94827279ccc97c3b5a05c8294b3b80ebde8f142e09ba4353ffbe91e4270e not found: ID does not exist"
Jul 24 18:09:42.657294 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.657260    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvpx6\" (UniqueName: \"kubernetes.io/projected/adde3481-ecda-4093-a808-ed32d7d2f668-kube-api-access-gvpx6\") pod \"adde3481-ecda-4093-a808-ed32d7d2f668\" (UID: \"adde3481-ecda-4093-a808-ed32d7d2f668\") "
Jul 24 18:09:42.657446 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.657315    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adde3481-ecda-4093-a808-ed32d7d2f668-openshift-service-ca\") pod \"adde3481-ecda-4093-a808-ed32d7d2f668\" (UID: \"adde3481-ecda-4093-a808-ed32d7d2f668\") "
Jul 24 18:09:42.657446 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.657394    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adde3481-ecda-4093-a808-ed32d7d2f668-maas-api-tls\") pod \"adde3481-ecda-4093-a808-ed32d7d2f668\" (UID: \"adde3481-ecda-4093-a808-ed32d7d2f668\") "
Jul 24 18:09:42.657446 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.657433    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83-plugins-config-volume\") pod \"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83\" (UID: \"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83\") "
Jul 24 18:09:42.657616 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.657472    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dxlk\" (UniqueName: \"kubernetes.io/projected/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83-kube-api-access-4dxlk\") pod \"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83\" (UID: \"baf8c4ce-5dfe-4847-a87b-ca1ff468ef83\") "
Jul 24 18:09:42.657777 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.657753    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adde3481-ecda-4093-a808-ed32d7d2f668-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adde3481-ecda-4093-a808-ed32d7d2f668" (UID: "adde3481-ecda-4093-a808-ed32d7d2f668"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:09:42.657886 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.657845    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "baf8c4ce-5dfe-4847-a87b-ca1ff468ef83" (UID: "baf8c4ce-5dfe-4847-a87b-ca1ff468ef83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:09:42.659448 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.659424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adde3481-ecda-4093-a808-ed32d7d2f668-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adde3481-ecda-4093-a808-ed32d7d2f668" (UID: "adde3481-ecda-4093-a808-ed32d7d2f668"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:09:42.659448 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.659435    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adde3481-ecda-4093-a808-ed32d7d2f668-kube-api-access-gvpx6" (OuterVolumeSpecName: "kube-api-access-gvpx6") pod "adde3481-ecda-4093-a808-ed32d7d2f668" (UID: "adde3481-ecda-4093-a808-ed32d7d2f668"). InnerVolumeSpecName "kube-api-access-gvpx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:09:42.659448 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.659428    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83-kube-api-access-4dxlk" (OuterVolumeSpecName: "kube-api-access-4dxlk") pod "baf8c4ce-5dfe-4847-a87b-ca1ff468ef83" (UID: "baf8c4ce-5dfe-4847-a87b-ca1ff468ef83"). InnerVolumeSpecName "kube-api-access-4dxlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:09:42.758813 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.758736    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adde3481-ecda-4093-a808-ed32d7d2f668-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:42.758813 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.758762    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:42.758813 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.758773    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dxlk\" (UniqueName: \"kubernetes.io/projected/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83-kube-api-access-4dxlk\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:42.758813 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.758781    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvpx6\" (UniqueName: \"kubernetes.io/projected/adde3481-ecda-4093-a808-ed32d7d2f668-kube-api-access-gvpx6\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:42.758813 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.758791    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adde3481-ecda-4093-a808-ed32d7d2f668-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:42.906495 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.906470    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"]
Jul 24 18:09:42.911046 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.911015    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b716847d-7fd755cbf-m5ldt"]
Jul 24 18:09:42.921003 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.920980    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"]
Jul 24 18:09:42.925257 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.925234    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b716847d-84dd5f79c8-vzdbf"]
Jul 24 18:09:42.961711 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.961678    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adde3481-ecda-4093-a808-ed32d7d2f668" path="/var/lib/kubelet/pods/adde3481-ecda-4093-a808-ed32d7d2f668/volumes"
Jul 24 18:09:42.962134 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:42.962115    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf8c4ce-5dfe-4847-a87b-ca1ff468ef83" path="/var/lib/kubelet/pods/baf8c4ce-5dfe-4847-a87b-ca1ff468ef83/volumes"
Jul 24 18:09:43.771027 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:43.770997    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:46.810015 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:46.809983    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:48.042586 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.042540    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"]
Jul 24 18:09:48.042965 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.042810    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" podUID="da464976-b8ce-4770-a7a7-83d645f42526" containerName="istio-proxy" containerID="cri-o://b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9" gracePeriod=30
Jul 24 18:09:48.069994 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.069962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:48.300092 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.300008    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:48.532157 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.532118    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:48.831141 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831114    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"]
Jul 24 18:09:48.831405 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831393    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerName="istio-proxy"
Jul 24 18:09:48.831451 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831406    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerName="istio-proxy"
Jul 24 18:09:48.831451 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831416    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerName="istio-proxy"
Jul 24 18:09:48.831451 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831421    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerName="istio-proxy"
Jul 24 18:09:48.831451 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831430    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adde3481-ecda-4093-a808-ed32d7d2f668" containerName="maas-api"
Jul 24 18:09:48.831451 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831435    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="adde3481-ecda-4093-a808-ed32d7d2f668" containerName="maas-api"
Jul 24 18:09:48.831451 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831441    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf8c4ce-5dfe-4847-a87b-ca1ff468ef83" containerName="payload-processing"
Jul 24 18:09:48.831451 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831447    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf8c4ce-5dfe-4847-a87b-ca1ff468ef83" containerName="payload-processing"
Jul 24 18:09:48.831680 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831495    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="adde3481-ecda-4093-a808-ed32d7d2f668" containerName="maas-api"
Jul 24 18:09:48.831680 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831503    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerName="istio-proxy"
Jul 24 18:09:48.831680 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831509    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf8c4ce-5dfe-4847-a87b-ca1ff468ef83" containerName="payload-processing"
Jul 24 18:09:48.831680 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.831516    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b84616-1fec-4e79-a32a-ff4ee81b994d" containerName="istio-proxy"
Jul 24 18:09:48.835328 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.835311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:48.838347 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.838330    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-7a6fbc6b-openshift-default-dockercfg-bggpp\""
Jul 24 18:09:48.847357 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.847331    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"]
Jul 24 18:09:48.869529 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:48.869503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:49.015075 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.015043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-data\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.015075 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.015074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-podinfo\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.015285 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.015099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-envoy\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.015285 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.015117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4de454e-5a97-457a-80d9-0bee81717d8c-istiod-ca-cert\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.015285 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.015213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-workload-socket\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.015285 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.015238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-token\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.015285 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.015273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjdvx\" (UniqueName: \"kubernetes.io/projected/d4de454e-5a97-457a-80d9-0bee81717d8c-kube-api-access-hjdvx\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.015536 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.015301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-credential-socket\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.015536 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.015364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-workload-certs\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.067034 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.067002    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:49.115832 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.115756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-credential-socket\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.115832 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.115798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-workload-certs\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.115832 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.115827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-data\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.116125 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.115854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-podinfo\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.116125 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.115884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-envoy\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.116125 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.115909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4de454e-5a97-457a-80d9-0bee81717d8c-istiod-ca-cert\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.116125 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.115959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-workload-socket\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.116125 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.115983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-token\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.116125 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.116032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdvx\" (UniqueName: \"kubernetes.io/projected/d4de454e-5a97-457a-80d9-0bee81717d8c-kube-api-access-hjdvx\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.116496 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.116467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-workload-certs\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.116590 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.116467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-credential-socket\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.116669 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.116645    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-workload-socket\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.116856 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.116833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-data\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.117634 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.117608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4de454e-5a97-457a-80d9-0bee81717d8c-istiod-ca-cert\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.122246 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.118598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-envoy\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.122246 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.119074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-podinfo\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.125473 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.125440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-token\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.125647 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.125477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdvx\" (UniqueName: \"kubernetes.io/projected/d4de454e-5a97-457a-80d9-0bee81717d8c-kube-api-access-hjdvx\") pod \"e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") " pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.144904 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.144884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:49.278669 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.278639    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"]
Jul 24 18:09:49.282322 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:09:49.282296    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4de454e_5a97_457a_80d9_0bee81717d8c.slice/crio-e49aacc8a363443d09eaa57f3eba09c50dce8dcc597182a96adba3282425f0cf WatchSource:0}: Error finding container e49aacc8a363443d09eaa57f3eba09c50dce8dcc597182a96adba3282425f0cf: Status 404 returned error can't find the container with id e49aacc8a363443d09eaa57f3eba09c50dce8dcc597182a96adba3282425f0cf
Jul 24 18:09:49.284334 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.284304    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:09:49.284431 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.284366    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:09:49.284431 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.284393    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:09:49.616306 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.616276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g" event={"ID":"d4de454e-5a97-457a-80d9-0bee81717d8c","Type":"ContainerStarted","Data":"fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc"}
Jul 24 18:09:49.616472 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:49.616326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g" event={"ID":"d4de454e-5a97-457a-80d9-0bee81717d8c","Type":"ContainerStarted","Data":"e49aacc8a363443d09eaa57f3eba09c50dce8dcc597182a96adba3282425f0cf"}
Jul 24 18:09:50.146009 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:50.145979    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:50.150977 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:50.150956    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:50.189110 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:50.189062    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g" podStartSLOduration=2.189046943 podStartE2EDuration="2.189046943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:09:49.641439393 +0000 UTC m=+2003.157950428" watchObservedRunningTime="2026-07-24 18:09:50.189046943 +0000 UTC m=+2003.705557978"
Jul 24 18:09:50.621035 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:50.621004    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:50.621876 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:50.621857    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:09:51.037634 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.037531    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:51.113948 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.113917    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"]
Jul 24 18:09:51.117784 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.117762    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:51.120241 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.120193    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-7a6fbc6b-serving-cert\""
Jul 24 18:09:51.120241 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.120206    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:09:51.120241 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.120225    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fvm85\""
Jul 24 18:09:51.121050 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.121028    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:09:51.129078 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.129058    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"]
Jul 24 18:09:51.150863 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.150835    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:51.236134 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.236103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37d448d-72a2-4a8f-b149-68968a9190ee-openshift-service-ca\") pod \"maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:51.236377 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.236344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh7nj\" (UniqueName: \"kubernetes.io/projected/f37d448d-72a2-4a8f-b149-68968a9190ee-kube-api-access-hh7nj\") pod \"maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:51.236493 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.236422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f37d448d-72a2-4a8f-b149-68968a9190ee-maas-api-tls\") pod \"maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:51.248998 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.248969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:51.337055 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.337030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f37d448d-72a2-4a8f-b149-68968a9190ee-maas-api-tls\") pod \"maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:51.337208 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:09:51.337190    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-serving-cert: secret "maas-api-e2e-derive-7a6fbc6b-serving-cert" not found
Jul 24 18:09:51.337260 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.337195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37d448d-72a2-4a8f-b149-68968a9190ee-openshift-service-ca\") pod \"maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:51.337260 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:09:51.337256    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f37d448d-72a2-4a8f-b149-68968a9190ee-maas-api-tls podName:f37d448d-72a2-4a8f-b149-68968a9190ee nodeName:}" failed. No retries permitted until 2026-07-24 18:09:51.837233443 +0000 UTC m=+2005.353744457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f37d448d-72a2-4a8f-b149-68968a9190ee-maas-api-tls") pod "maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85" (UID: "f37d448d-72a2-4a8f-b149-68968a9190ee") : secret "maas-api-e2e-derive-7a6fbc6b-serving-cert" not found
Jul 24 18:09:51.337357 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.337285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh7nj\" (UniqueName: \"kubernetes.io/projected/f37d448d-72a2-4a8f-b149-68968a9190ee-kube-api-access-hh7nj\") pod \"maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:51.337779 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.337758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37d448d-72a2-4a8f-b149-68968a9190ee-openshift-service-ca\") pod \"maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:51.345957 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.345938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh7nj\" (UniqueName: \"kubernetes.io/projected/f37d448d-72a2-4a8f-b149-68968a9190ee-kube-api-access-hh7nj\") pod \"maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:51.634399 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.634309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:51.739034 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.739003    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:51.841433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.841400    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"]
Jul 24 18:09:51.842967 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.842946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f37d448d-72a2-4a8f-b149-68968a9190ee-maas-api-tls\") pod \"maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:51.843993 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.843971    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:09:51.844690 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.844669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"
Jul 24 18:09:51.845527 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.845493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f37d448d-72a2-4a8f-b149-68968a9190ee-maas-api-tls\") pod \"maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:51.847075 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.847055    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-7a6fbc6b\""
Jul 24 18:09:51.847243 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.847227    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-7a6fbc6b-dockercfg-8bk9b\""
Jul 24 18:09:51.855415 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.855387    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"]
Jul 24 18:09:51.944075 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.943991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj\" (UID: \"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"
Jul 24 18:09:51.944075 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:51.944048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvljs\" (UniqueName: \"kubernetes.io/projected/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf-kube-api-access-lvljs\") pod \"payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj\" (UID: \"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"
Jul 24 18:09:52.029286 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.029256    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:52.044619 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.044590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj\" (UID: \"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"
Jul 24 18:09:52.044741 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.044668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvljs\" (UniqueName: \"kubernetes.io/projected/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf-kube-api-access-lvljs\") pod \"payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj\" (UID: \"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"
Jul 24 18:09:52.045442 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.045417    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj\" (UID: \"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"
Jul 24 18:09:52.054037 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.054013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvljs\" (UniqueName: \"kubernetes.io/projected/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf-kube-api-access-lvljs\") pod \"payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj\" (UID: \"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"
Jul 24 18:09:52.154641 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.154608    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"]
Jul 24 18:09:52.157476 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:09:52.157451    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37d448d_72a2_4a8f_b149_68968a9190ee.slice/crio-c563ea98a9e88f6336ab8e9b21003b3bcc79b0176adae4fc7673c432241851ca WatchSource:0}: Error finding container c563ea98a9e88f6336ab8e9b21003b3bcc79b0176adae4fc7673c432241851ca: Status 404 returned error can't find the container with id c563ea98a9e88f6336ab8e9b21003b3bcc79b0176adae4fc7673c432241851ca
Jul 24 18:09:52.163476 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.163454    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"
Jul 24 18:09:52.288753 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.288724    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"]
Jul 24 18:09:52.291367 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:09:52.291337    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdd294a_f2dd_4ae4_8773_b8bd6f0c06cf.slice/crio-1e2791808e9c1794936789e3fdae8040373fecd286ae87e64fd7507434d8a2c3 WatchSource:0}: Error finding container 1e2791808e9c1794936789e3fdae8040373fecd286ae87e64fd7507434d8a2c3: Status 404 returned error can't find the container with id 1e2791808e9c1794936789e3fdae8040373fecd286ae87e64fd7507434d8a2c3
Jul 24 18:09:52.636710 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.636682    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj" event={"ID":"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf","Type":"ContainerStarted","Data":"f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828"}
Jul 24 18:09:52.636838 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.636767    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj" event={"ID":"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf","Type":"ContainerStarted","Data":"1e2791808e9c1794936789e3fdae8040373fecd286ae87e64fd7507434d8a2c3"}
Jul 24 18:09:52.636838 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.636801    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"
Jul 24 18:09:52.638114 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.638092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85" event={"ID":"f37d448d-72a2-4a8f-b149-68968a9190ee","Type":"ContainerStarted","Data":"bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913"}
Jul 24 18:09:52.638208 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.638129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85" event={"ID":"f37d448d-72a2-4a8f-b149-68968a9190ee","Type":"ContainerStarted","Data":"c563ea98a9e88f6336ab8e9b21003b3bcc79b0176adae4fc7673c432241851ca"}
Jul 24 18:09:52.655004 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.654946    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj" podStartSLOduration=1.654926157 podStartE2EDuration="1.654926157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:09:52.653379663 +0000 UTC m=+2006.169890698" watchObservedRunningTime="2026-07-24 18:09:52.654926157 +0000 UTC m=+2006.171437173"
Jul 24 18:09:52.671147 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:52.671091    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85" podStartSLOduration=1.254793248 podStartE2EDuration="1.671070673s" totalImagesPullingTime="416.277425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:09:52.670562714 +0000 UTC m=+2006.187073749" watchObservedRunningTime="2026-07-24 18:09:52.671070673 +0000 UTC m=+2006.187581712"
Jul 24 18:09:53.285957 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.285934    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:53.457515 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.457479    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99nd9\" (UniqueName: \"kubernetes.io/projected/da464976-b8ce-4770-a7a7-83d645f42526-kube-api-access-99nd9\") pod \"da464976-b8ce-4770-a7a7-83d645f42526\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") "
Jul 24 18:09:53.457734 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.457554    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-istio-data\") pod \"da464976-b8ce-4770-a7a7-83d645f42526\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") "
Jul 24 18:09:53.457734 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.457608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da464976-b8ce-4770-a7a7-83d645f42526-istiod-ca-cert\") pod \"da464976-b8ce-4770-a7a7-83d645f42526\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") "
Jul 24 18:09:53.457734 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.457641    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-workload-certs\") pod \"da464976-b8ce-4770-a7a7-83d645f42526\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") "
Jul 24 18:09:53.457734 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.457672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da464976-b8ce-4770-a7a7-83d645f42526-istio-token\") pod \"da464976-b8ce-4770-a7a7-83d645f42526\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") "
Jul 24 18:09:53.457734 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.457704    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-credential-socket\") pod \"da464976-b8ce-4770-a7a7-83d645f42526\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") "
Jul 24 18:09:53.457734 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.457736    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-workload-socket\") pod \"da464976-b8ce-4770-a7a7-83d645f42526\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") "
Jul 24 18:09:53.458060 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.457821    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da464976-b8ce-4770-a7a7-83d645f42526-istio-podinfo\") pod \"da464976-b8ce-4770-a7a7-83d645f42526\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") "
Jul 24 18:09:53.458060 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.457860    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-istio-envoy\") pod \"da464976-b8ce-4770-a7a7-83d645f42526\" (UID: \"da464976-b8ce-4770-a7a7-83d645f42526\") "
Jul 24 18:09:53.458168 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.458091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da464976-b8ce-4770-a7a7-83d645f42526" (UID: "da464976-b8ce-4770-a7a7-83d645f42526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:09:53.458227 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.458185    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da464976-b8ce-4770-a7a7-83d645f42526-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da464976-b8ce-4770-a7a7-83d645f42526" (UID: "da464976-b8ce-4770-a7a7-83d645f42526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:09:53.458284 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.458263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da464976-b8ce-4770-a7a7-83d645f42526" (UID: "da464976-b8ce-4770-a7a7-83d645f42526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:09:53.458423 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.458396    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da464976-b8ce-4770-a7a7-83d645f42526" (UID: "da464976-b8ce-4770-a7a7-83d645f42526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:09:53.458639 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.458610    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-istio-data" (OuterVolumeSpecName: "istio-data") pod "da464976-b8ce-4770-a7a7-83d645f42526" (UID: "da464976-b8ce-4770-a7a7-83d645f42526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:09:53.461786 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.461705    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da464976-b8ce-4770-a7a7-83d645f42526-istio-token" (OuterVolumeSpecName: "istio-token") pod "da464976-b8ce-4770-a7a7-83d645f42526" (UID: "da464976-b8ce-4770-a7a7-83d645f42526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:09:53.461786 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.461705    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da464976-b8ce-4770-a7a7-83d645f42526" (UID: "da464976-b8ce-4770-a7a7-83d645f42526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:09:53.461786 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.461725    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da464976-b8ce-4770-a7a7-83d645f42526-kube-api-access-99nd9" (OuterVolumeSpecName: "kube-api-access-99nd9") pod "da464976-b8ce-4770-a7a7-83d645f42526" (UID: "da464976-b8ce-4770-a7a7-83d645f42526"). InnerVolumeSpecName "kube-api-access-99nd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:09:53.462010 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.461964    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da464976-b8ce-4770-a7a7-83d645f42526-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da464976-b8ce-4770-a7a7-83d645f42526" (UID: "da464976-b8ce-4770-a7a7-83d645f42526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 18:09:53.558885 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.558848    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:53.558885 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.558885    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da464976-b8ce-4770-a7a7-83d645f42526-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:53.559076 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.558899    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:53.559076 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.558913    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da464976-b8ce-4770-a7a7-83d645f42526-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:53.559076 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.558926    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:53.559076 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.558939    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:53.559076 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.558952    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da464976-b8ce-4770-a7a7-83d645f42526-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:53.559076 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.558965    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da464976-b8ce-4770-a7a7-83d645f42526-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:53.559076 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.558977    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99nd9\" (UniqueName: \"kubernetes.io/projected/da464976-b8ce-4770-a7a7-83d645f42526-kube-api-access-99nd9\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:09:53.644534 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.644498    2581 generic.go:356] "Generic (PLEG): container finished" podID="da464976-b8ce-4770-a7a7-83d645f42526" containerID="b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9" exitCode=0
Jul 24 18:09:53.644708 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.644556    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"
Jul 24 18:09:53.644708 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.644603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" event={"ID":"da464976-b8ce-4770-a7a7-83d645f42526","Type":"ContainerDied","Data":"b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9"}
Jul 24 18:09:53.644708 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.644671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x" event={"ID":"da464976-b8ce-4770-a7a7-83d645f42526","Type":"ContainerDied","Data":"f2cae3d871f508daf3d0367fa0c434fa9d86bb98d6f0cdcebea272549aed59e7"}
Jul 24 18:09:53.644708 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.644706    2581 scope.go:117] "RemoveContainer" containerID="b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9"
Jul 24 18:09:53.645154 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.645135    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:09:53.654033 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.654017    2581 scope.go:117] "RemoveContainer" containerID="0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2"
Jul 24 18:09:53.661536 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.661506    2581 scope.go:117] "RemoveContainer" containerID="b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9"
Jul 24 18:09:53.661809 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:09:53.661784    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9\": container with ID starting with b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9 not found: ID does not exist" containerID="b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9"
Jul 24 18:09:53.661879 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.661819    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9"} err="failed to get container status \"b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9\": rpc error: code = NotFound desc = could not find container \"b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9\": container with ID starting with b499cf04ec1d967e8489ce7b7256a012deabe11ceeefbca8cfdc3bf623974ad9 not found: ID does not exist"
Jul 24 18:09:53.661879 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.661837    2581 scope.go:117] "RemoveContainer" containerID="0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2"
Jul 24 18:09:53.662090 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:09:53.662070    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2\": container with ID starting with 0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2 not found: ID does not exist" containerID="0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2"
Jul 24 18:09:53.662139 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.662096    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2"} err="failed to get container status \"0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2\": rpc error: code = NotFound desc = could not find container \"0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2\": container with ID starting with 0bbe436d570f72aaa45a99d1e310d6e3011c4e030f8b32561b8333c5d497d8d2 not found: ID does not exist"
Jul 24 18:09:53.669192 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.669170    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"]
Jul 24 18:09:53.675470 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:53.675450    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b716847d-openshift-default-6c587c6bf8-s898x"]
Jul 24 18:09:54.958354 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:54.958321    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da464976-b8ce-4770-a7a7-83d645f42526" path="/var/lib/kubelet/pods/da464976-b8ce-4770-a7a7-83d645f42526/volumes"
Jul 24 18:09:59.665584 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:09:59.665540    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:10:03.647630 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:03.647601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"
Jul 24 18:10:04.705729 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:04.705694    2581 generic.go:356] "Generic (PLEG): container finished" podID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerID="fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc" exitCode=0
Jul 24 18:10:04.706141 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:04.705765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g" event={"ID":"d4de454e-5a97-457a-80d9-0bee81717d8c","Type":"ContainerDied","Data":"fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc"}
Jul 24 18:10:04.706141 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:04.706120    2581 scope.go:117] "RemoveContainer" containerID="fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc"
Jul 24 18:10:04.707292 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:04.707263    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:10:04.707365 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:04.707348    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:10:04.707415 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:04.707399    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:10:05.621979 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:05.621947    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:10:05.711896 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:05.711865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g" event={"ID":"d4de454e-5a97-457a-80d9-0bee81717d8c","Type":"ContainerStarted","Data":"c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8"}
Jul 24 18:10:06.146363 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:06.146330    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:10:06.146761 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:06.146736    2581 patch_prober.go:28] interesting pod/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 24 18:10:06.146949 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:06.146922    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g" podUID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 24 18:10:07.146191 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:07.146146    2581 patch_prober.go:28] interesting pod/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 24 18:10:07.146705 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:07.146215    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g" podUID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 24 18:10:08.146291 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:08.146254    2581 patch_prober.go:28] interesting pod/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 24 18:10:08.146927 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:08.146896    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g" podUID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 24 18:10:09.145431 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:09.145396    2581 patch_prober.go:28] interesting pod/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 24 18:10:09.145751 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:09.145721    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g" podUID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 24 18:10:10.149147 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:10.149116    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:10:10.149514 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:10.149461    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:10:10.150071 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:10.150053    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:10:15.951599 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:15.951557    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x"]
Jul 24 18:10:15.951979 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:15.951860    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da464976-b8ce-4770-a7a7-83d645f42526" containerName="istio-proxy"
Jul 24 18:10:15.951979 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:15.951870    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da464976-b8ce-4770-a7a7-83d645f42526" containerName="istio-proxy"
Jul 24 18:10:15.951979 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:15.951879    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da464976-b8ce-4770-a7a7-83d645f42526" containerName="istio-proxy"
Jul 24 18:10:15.951979 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:15.951885    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da464976-b8ce-4770-a7a7-83d645f42526" containerName="istio-proxy"
Jul 24 18:10:15.951979 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:15.951931    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da464976-b8ce-4770-a7a7-83d645f42526" containerName="istio-proxy"
Jul 24 18:10:15.951979 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:15.951938    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da464976-b8ce-4770-a7a7-83d645f42526" containerName="istio-proxy"
Jul 24 18:10:15.955166 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:15.955148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x"
Jul 24 18:10:15.957584 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:15.957548    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v7cm8\""
Jul 24 18:10:15.962603 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:15.962258    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x"]
Jul 24 18:10:16.035268 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:16.035238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b1f015-6b36-4ae0-89d6-e4fa7520ed9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x\" (UID: \"43b1f015-6b36-4ae0-89d6-e4fa7520ed9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x"
Jul 24 18:10:16.135755 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:16.135721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b1f015-6b36-4ae0-89d6-e4fa7520ed9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x\" (UID: \"43b1f015-6b36-4ae0-89d6-e4fa7520ed9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x"
Jul 24 18:10:16.136282 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:16.136264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b1f015-6b36-4ae0-89d6-e4fa7520ed9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x\" (UID: \"43b1f015-6b36-4ae0-89d6-e4fa7520ed9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x"
Jul 24 18:10:16.267402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:16.267325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x"
Jul 24 18:10:16.601273 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:16.601250    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x"]
Jul 24 18:10:16.603036 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:10:16.603006    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b1f015_6b36_4ae0_89d6_e4fa7520ed9f.slice/crio-a994d2c4f874677000f3212f60ff610f8a77308c3d9f21658e6e059519badb5e WatchSource:0}: Error finding container a994d2c4f874677000f3212f60ff610f8a77308c3d9f21658e6e059519badb5e: Status 404 returned error can't find the container with id a994d2c4f874677000f3212f60ff610f8a77308c3d9f21658e6e059519badb5e
Jul 24 18:10:16.755840 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:16.755809    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x" event={"ID":"43b1f015-6b36-4ae0-89d6-e4fa7520ed9f","Type":"ContainerStarted","Data":"a994d2c4f874677000f3212f60ff610f8a77308c3d9f21658e6e059519badb5e"}
Jul 24 18:10:17.761937 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:17.761897    2581 generic.go:356] "Generic (PLEG): container finished" podID="43b1f015-6b36-4ae0-89d6-e4fa7520ed9f" containerID="bd34ccfdd66cb3dd596ac88e38fd0a6631b31a3b0e0203078bac14635cc5cd07" exitCode=0
Jul 24 18:10:17.762377 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:17.761968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x" event={"ID":"43b1f015-6b36-4ae0-89d6-e4fa7520ed9f","Type":"ContainerDied","Data":"bd34ccfdd66cb3dd596ac88e38fd0a6631b31a3b0e0203078bac14635cc5cd07"}
Jul 24 18:10:18.887453 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:18.887431    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x"
Jul 24 18:10:18.959747 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:18.959719    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b1f015-6b36-4ae0-89d6-e4fa7520ed9f-maas-api-service-ca\") pod \"43b1f015-6b36-4ae0-89d6-e4fa7520ed9f\" (UID: \"43b1f015-6b36-4ae0-89d6-e4fa7520ed9f\") "
Jul 24 18:10:18.960037 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:18.960014    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b1f015-6b36-4ae0-89d6-e4fa7520ed9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43b1f015-6b36-4ae0-89d6-e4fa7520ed9f" (UID: "43b1f015-6b36-4ae0-89d6-e4fa7520ed9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:10:19.061180 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:19.061151    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b1f015-6b36-4ae0-89d6-e4fa7520ed9f-maas-api-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:19.772178 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:19.772146    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x"
Jul 24 18:10:19.772178 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:19.772167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x" event={"ID":"43b1f015-6b36-4ae0-89d6-e4fa7520ed9f","Type":"ContainerDied","Data":"a994d2c4f874677000f3212f60ff610f8a77308c3d9f21658e6e059519badb5e"}
Jul 24 18:10:19.772495 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:19.772221    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a994d2c4f874677000f3212f60ff610f8a77308c3d9f21658e6e059519badb5e"
Jul 24 18:10:26.401188 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.401140    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"]
Jul 24 18:10:26.401665 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.401486    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85" podUID="f37d448d-72a2-4a8f-b149-68968a9190ee" containerName="maas-api" containerID="cri-o://bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913" gracePeriod=30
Jul 24 18:10:26.423187 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.423164    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:26.459608 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.459561    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"]
Jul 24 18:10:26.459841 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.459798    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj" podUID="efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf" containerName="payload-pre-processing" containerID="cri-o://f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828" gracePeriod=30
Jul 24 18:10:26.487309 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.487283    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:26.587500 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.587450    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:26.728553 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.728529    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:10:26.731433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.731411    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"
Jul 24 18:10:26.803948 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.803918    2581 generic.go:356] "Generic (PLEG): container finished" podID="efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf" containerID="f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828" exitCode=0
Jul 24 18:10:26.804106 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.803958    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"
Jul 24 18:10:26.804106 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.804001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj" event={"ID":"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf","Type":"ContainerDied","Data":"f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828"}
Jul 24 18:10:26.804106 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.804053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj" event={"ID":"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf","Type":"ContainerDied","Data":"1e2791808e9c1794936789e3fdae8040373fecd286ae87e64fd7507434d8a2c3"}
Jul 24 18:10:26.804106 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.804075    2581 scope.go:117] "RemoveContainer" containerID="f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828"
Jul 24 18:10:26.805289 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.805270    2581 generic.go:356] "Generic (PLEG): container finished" podID="f37d448d-72a2-4a8f-b149-68968a9190ee" containerID="bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913" exitCode=0
Jul 24 18:10:26.805407 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.805314    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"
Jul 24 18:10:26.805407 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.805356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85" event={"ID":"f37d448d-72a2-4a8f-b149-68968a9190ee","Type":"ContainerDied","Data":"bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913"}
Jul 24 18:10:26.805407 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.805396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85" event={"ID":"f37d448d-72a2-4a8f-b149-68968a9190ee","Type":"ContainerDied","Data":"c563ea98a9e88f6336ab8e9b21003b3bcc79b0176adae4fc7673c432241851ca"}
Jul 24 18:10:26.812110 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.812092    2581 scope.go:117] "RemoveContainer" containerID="f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828"
Jul 24 18:10:26.812360 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:10:26.812339    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828\": container with ID starting with f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828 not found: ID does not exist" containerID="f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828"
Jul 24 18:10:26.812438 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.812371    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828"} err="failed to get container status \"f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828\": rpc error: code = NotFound desc = could not find container \"f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828\": container with ID starting with f8d370c0152109e20fd430a20a3c042e6e574d6c28bccd262ca218b967856828 not found: ID does not exist"
Jul 24 18:10:26.812438 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.812395    2581 scope.go:117] "RemoveContainer" containerID="bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913"
Jul 24 18:10:26.819665 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.819648    2581 scope.go:117] "RemoveContainer" containerID="bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913"
Jul 24 18:10:26.819923 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:10:26.819903    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913\": container with ID starting with bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913 not found: ID does not exist" containerID="bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913"
Jul 24 18:10:26.819971 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.819928    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913"} err="failed to get container status \"bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913\": rpc error: code = NotFound desc = could not find container \"bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913\": container with ID starting with bc79a3c257fbdca5fdef5e6743f3331412cbb76e13d919dc1a726060a1ee7913 not found: ID does not exist"
Jul 24 18:10:26.827193 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.827176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh7nj\" (UniqueName: \"kubernetes.io/projected/f37d448d-72a2-4a8f-b149-68968a9190ee-kube-api-access-hh7nj\") pod \"f37d448d-72a2-4a8f-b149-68968a9190ee\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") "
Jul 24 18:10:26.827268 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.827257    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvljs\" (UniqueName: \"kubernetes.io/projected/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf-kube-api-access-lvljs\") pod \"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf\" (UID: \"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf\") "
Jul 24 18:10:26.827305 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.827278    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf-plugins-config-volume\") pod \"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf\" (UID: \"efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf\") "
Jul 24 18:10:26.827338 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.827311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37d448d-72a2-4a8f-b149-68968a9190ee-openshift-service-ca\") pod \"f37d448d-72a2-4a8f-b149-68968a9190ee\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") "
Jul 24 18:10:26.827338 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.827329    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f37d448d-72a2-4a8f-b149-68968a9190ee-maas-api-tls\") pod \"f37d448d-72a2-4a8f-b149-68968a9190ee\" (UID: \"f37d448d-72a2-4a8f-b149-68968a9190ee\") "
Jul 24 18:10:26.827673 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.827651    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf" (UID: "efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:10:26.827673 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.827652    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f37d448d-72a2-4a8f-b149-68968a9190ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f37d448d-72a2-4a8f-b149-68968a9190ee" (UID: "f37d448d-72a2-4a8f-b149-68968a9190ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:10:26.829150 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.829127    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f37d448d-72a2-4a8f-b149-68968a9190ee-kube-api-access-hh7nj" (OuterVolumeSpecName: "kube-api-access-hh7nj") pod "f37d448d-72a2-4a8f-b149-68968a9190ee" (UID: "f37d448d-72a2-4a8f-b149-68968a9190ee"). InnerVolumeSpecName "kube-api-access-hh7nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:10:26.829240 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.829225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf-kube-api-access-lvljs" (OuterVolumeSpecName: "kube-api-access-lvljs") pod "efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf" (UID: "efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf"). InnerVolumeSpecName "kube-api-access-lvljs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:10:26.829240 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.829230    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f37d448d-72a2-4a8f-b149-68968a9190ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f37d448d-72a2-4a8f-b149-68968a9190ee" (UID: "f37d448d-72a2-4a8f-b149-68968a9190ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:10:26.928625 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.928543    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvljs\" (UniqueName: \"kubernetes.io/projected/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf-kube-api-access-lvljs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:26.928625 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.928565    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:26.928625 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.928593    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37d448d-72a2-4a8f-b149-68968a9190ee-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:26.928625 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.928604    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f37d448d-72a2-4a8f-b149-68968a9190ee-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:26.928625 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:26.928616    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh7nj\" (UniqueName: \"kubernetes.io/projected/f37d448d-72a2-4a8f-b149-68968a9190ee-kube-api-access-hh7nj\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:27.120111 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:27.120082    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"]
Jul 24 18:10:27.124623 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:27.124598    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7a6fbc6b-f9787f4dd-f77sj"]
Jul 24 18:10:27.134498 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:27.134480    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"]
Jul 24 18:10:27.138267 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:27.138246    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7a6fbc6b-5f8b9c6bb8-r2d85"]
Jul 24 18:10:28.957417 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:28.957382    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf" path="/var/lib/kubelet/pods/efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf/volumes"
Jul 24 18:10:28.957804 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:28.957785    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f37d448d-72a2-4a8f-b149-68968a9190ee" path="/var/lib/kubelet/pods/f37d448d-72a2-4a8f-b149-68968a9190ee/volumes"
Jul 24 18:10:31.064350 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:31.064314    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x"]
Jul 24 18:10:31.067942 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:31.067913    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a6fbc6b-qsj8x"]
Jul 24 18:10:31.074943 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:31.074921    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:31.195482 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:31.195450    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"]
Jul 24 18:10:31.195704 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:31.195676    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g" podUID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerName="istio-proxy" containerID="cri-o://c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8" gracePeriod=30
Jul 24 18:10:31.228888 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:31.228859    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:31.473745 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:31.473665    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:32.028462 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:32.028429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:32.137289 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:32.137254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:32.247940 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:32.247911    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:32.957771 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:32.957737    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43b1f015-6b36-4ae0-89d6-e4fa7520ed9f" path="/var/lib/kubelet/pods/43b1f015-6b36-4ae0-89d6-e4fa7520ed9f/volumes"
Jul 24 18:10:33.658463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:33.658429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:36.439391 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.439367    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:10:36.514024 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.513945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-token\") pod \"d4de454e-5a97-457a-80d9-0bee81717d8c\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") "
Jul 24 18:10:36.514024 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.513983    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-workload-socket\") pod \"d4de454e-5a97-457a-80d9-0bee81717d8c\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") "
Jul 24 18:10:36.514024 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514022    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-podinfo\") pod \"d4de454e-5a97-457a-80d9-0bee81717d8c\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") "
Jul 24 18:10:36.514295 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514079    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjdvx\" (UniqueName: \"kubernetes.io/projected/d4de454e-5a97-457a-80d9-0bee81717d8c-kube-api-access-hjdvx\") pod \"d4de454e-5a97-457a-80d9-0bee81717d8c\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") "
Jul 24 18:10:36.514295 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514108    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4de454e-5a97-457a-80d9-0bee81717d8c-istiod-ca-cert\") pod \"d4de454e-5a97-457a-80d9-0bee81717d8c\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") "
Jul 24 18:10:36.514295 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514130    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-workload-certs\") pod \"d4de454e-5a97-457a-80d9-0bee81717d8c\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") "
Jul 24 18:10:36.514295 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514167    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-credential-socket\") pod \"d4de454e-5a97-457a-80d9-0bee81717d8c\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") "
Jul 24 18:10:36.514295 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514194    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-envoy\") pod \"d4de454e-5a97-457a-80d9-0bee81717d8c\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") "
Jul 24 18:10:36.514295 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514233    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-data\") pod \"d4de454e-5a97-457a-80d9-0bee81717d8c\" (UID: \"d4de454e-5a97-457a-80d9-0bee81717d8c\") "
Jul 24 18:10:36.514295 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514230    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d4de454e-5a97-457a-80d9-0bee81717d8c" (UID: "d4de454e-5a97-457a-80d9-0bee81717d8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:10:36.514671 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d4de454e-5a97-457a-80d9-0bee81717d8c" (UID: "d4de454e-5a97-457a-80d9-0bee81717d8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:10:36.514671 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514449    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d4de454e-5a97-457a-80d9-0bee81717d8c" (UID: "d4de454e-5a97-457a-80d9-0bee81717d8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:10:36.514671 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514509    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4de454e-5a97-457a-80d9-0bee81717d8c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d4de454e-5a97-457a-80d9-0bee81717d8c" (UID: "d4de454e-5a97-457a-80d9-0bee81717d8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:10:36.514671 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514548    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:36.514671 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514589    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:36.514671 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.514605    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:36.515189 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.515167    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d4de454e-5a97-457a-80d9-0bee81717d8c" (UID: "d4de454e-5a97-457a-80d9-0bee81717d8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:10:36.516251 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.516230    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4de454e-5a97-457a-80d9-0bee81717d8c-kube-api-access-hjdvx" (OuterVolumeSpecName: "kube-api-access-hjdvx") pod "d4de454e-5a97-457a-80d9-0bee81717d8c" (UID: "d4de454e-5a97-457a-80d9-0bee81717d8c"). InnerVolumeSpecName "kube-api-access-hjdvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:10:36.516413 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.516386    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d4de454e-5a97-457a-80d9-0bee81717d8c" (UID: "d4de454e-5a97-457a-80d9-0bee81717d8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:10:36.516645 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.516620    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d4de454e-5a97-457a-80d9-0bee81717d8c" (UID: "d4de454e-5a97-457a-80d9-0bee81717d8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:10:36.516746 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.516642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d4de454e-5a97-457a-80d9-0bee81717d8c" (UID: "d4de454e-5a97-457a-80d9-0bee81717d8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 18:10:36.615917 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.615884    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:36.615917 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.615910    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:36.615917 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.615918    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:36.616130 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.615926    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4de454e-5a97-457a-80d9-0bee81717d8c-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:36.616130 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.615936    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjdvx\" (UniqueName: \"kubernetes.io/projected/d4de454e-5a97-457a-80d9-0bee81717d8c-kube-api-access-hjdvx\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:36.616130 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.615946    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4de454e-5a97-457a-80d9-0bee81717d8c-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:10:36.853810 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.853769    2581 generic.go:356] "Generic (PLEG): container finished" podID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerID="c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8" exitCode=0
Jul 24 18:10:36.853810 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.853812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g" event={"ID":"d4de454e-5a97-457a-80d9-0bee81717d8c","Type":"ContainerDied","Data":"c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8"}
Jul 24 18:10:36.854059 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.853821    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"
Jul 24 18:10:36.854059 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.853862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g" event={"ID":"d4de454e-5a97-457a-80d9-0bee81717d8c","Type":"ContainerDied","Data":"e49aacc8a363443d09eaa57f3eba09c50dce8dcc597182a96adba3282425f0cf"}
Jul 24 18:10:36.854059 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.853887    2581 scope.go:117] "RemoveContainer" containerID="c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8"
Jul 24 18:10:36.864462 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.864438    2581 scope.go:117] "RemoveContainer" containerID="fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc"
Jul 24 18:10:36.872212 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.872194    2581 scope.go:117] "RemoveContainer" containerID="c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8"
Jul 24 18:10:36.872458 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:10:36.872437    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8\": container with ID starting with c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8 not found: ID does not exist" containerID="c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8"
Jul 24 18:10:36.872531 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.872470    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8"} err="failed to get container status \"c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8\": rpc error: code = NotFound desc = could not find container \"c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8\": container with ID starting with c98e8050a264ed7de5756035712273c7bd7e7816134fbefeb083446e65432ec8 not found: ID does not exist"
Jul 24 18:10:36.872531 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.872493    2581 scope.go:117] "RemoveContainer" containerID="fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc"
Jul 24 18:10:36.872768 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:10:36.872750    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc\": container with ID starting with fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc not found: ID does not exist" containerID="fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc"
Jul 24 18:10:36.872823 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.872774    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc"} err="failed to get container status \"fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc\": rpc error: code = NotFound desc = could not find container \"fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc\": container with ID starting with fbf9bfa3a339af5576b8e04cbd2bb5946341070857a73e82a8a0492b4ac695bc not found: ID does not exist"
Jul 24 18:10:36.879133 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.879109    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"]
Jul 24 18:10:36.884466 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.884443    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-7a6fbc6b-openshift-default-7bbb77f4cb-2dx7g"]
Jul 24 18:10:36.957178 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:36.957153    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4de454e-5a97-457a-80d9-0bee81717d8c" path="/var/lib/kubelet/pods/d4de454e-5a97-457a-80d9-0bee81717d8c/volumes"
Jul 24 18:10:39.934197 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:39.934166    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:40.054767 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:40.054738    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:40.151957 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:40.151924    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:40.253137 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:40.253044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:10:40.694585 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:10:40.694540    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:06.723307 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:06.723209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:13.360306 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:13.360272    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:13.468339 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:13.468309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:13.563934 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:13.563897    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:29.557343 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:29.557301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:35.271802 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:35.271768    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:35.446584 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:35.446544    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:35.591258 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:35.591230    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:46.047376 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:46.047317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:46.862560 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:46.862524    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:53.410399 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:53.410364    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:53.513947 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:53.513910    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:53.610905 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:53.610873    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:53.711062 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:53.710985    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:54.578252 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:54.578221    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:54.675037 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:54.675007    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:54.773250 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:54.773207    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:11:56.494130 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:11:56.494083    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:04.803685 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:04.803642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:04.937655 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:04.937621    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:05.502042 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:05.502014    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:08.924025 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:08.923992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:12.997115 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:12.997085    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:13.087821 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.087791    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:13.223685 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.223654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:13.586197 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586163    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"]
Jul 24 18:12:13.586435 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586424    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerName="istio-proxy"
Jul 24 18:12:13.586480 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586437    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerName="istio-proxy"
Jul 24 18:12:13.586480 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586445    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b1f015-6b36-4ae0-89d6-e4fa7520ed9f" containerName="revoke-keys"
Jul 24 18:12:13.586480 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586451    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b1f015-6b36-4ae0-89d6-e4fa7520ed9f" containerName="revoke-keys"
Jul 24 18:12:13.586480 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586458    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf" containerName="payload-pre-processing"
Jul 24 18:12:13.586480 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586464    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf" containerName="payload-pre-processing"
Jul 24 18:12:13.586480 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586477    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f37d448d-72a2-4a8f-b149-68968a9190ee" containerName="maas-api"
Jul 24 18:12:13.586480 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586482    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f37d448d-72a2-4a8f-b149-68968a9190ee" containerName="maas-api"
Jul 24 18:12:13.586725 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586527    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerName="istio-proxy"
Jul 24 18:12:13.586725 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586536    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="efdd294a-f2dd-4ae4-8773-b8bd6f0c06cf" containerName="payload-pre-processing"
Jul 24 18:12:13.586725 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586544    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b1f015-6b36-4ae0-89d6-e4fa7520ed9f" containerName="revoke-keys"
Jul 24 18:12:13.586725 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.586549    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f37d448d-72a2-4a8f-b149-68968a9190ee" containerName="maas-api"
Jul 24 18:12:13.589364 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.589341    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.592625 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.592605    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ba2e7e10-openshift-default-dockercfg-x9cr8\""
Jul 24 18:12:13.626279 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.626248    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"]
Jul 24 18:12:13.649583 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.649538    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:13.665331 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.665298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-envoy\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.665460 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.665341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-credential-socket\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.665460 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.665373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-podinfo\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.665460 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.665403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whzct\" (UniqueName: \"kubernetes.io/projected/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-kube-api-access-whzct\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.665460 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.665441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-token\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.665710 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.665466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istiod-ca-cert\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.665710 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.665492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-data\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.665710 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.665558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-workload-socket\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.665710 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.665620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-workload-certs\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.750751 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.750715    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:13.766413 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.766386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-envoy\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.766559 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.766424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-credential-socket\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.766559 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.766455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-podinfo\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.766559 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.766481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whzct\" (UniqueName: \"kubernetes.io/projected/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-kube-api-access-whzct\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.766559 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.766520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-token\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.766559 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.766542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istiod-ca-cert\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.766839 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.766583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-data\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.766839 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.766641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-workload-socket\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.766839 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.766676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-workload-certs\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.766995 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.766867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-credential-socket\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.766995 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.766986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-data\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.767111 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.767080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-workload-socket\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.767183 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.767162    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istiod-ca-cert\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.767307 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.767290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-workload-certs\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.768810 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.768792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-envoy\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.768913 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.768897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-podinfo\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.777649 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.777627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-token\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.777823 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.777807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whzct\" (UniqueName: \"kubernetes.io/projected/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-kube-api-access-whzct\") pod \"e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") " pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:13.899381 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:13.899305    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:14.029672 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:14.029634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"]
Jul 24 18:12:14.033436 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:12:14.033411    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fef7c5_2e83_4137_ae26_84e22d1b41a9.slice/crio-7bb37b05d9e764f05e158ba05c418202124ff067ef018bbc621579646cde8994 WatchSource:0}: Error finding container 7bb37b05d9e764f05e158ba05c418202124ff067ef018bbc621579646cde8994: Status 404 returned error can't find the container with id 7bb37b05d9e764f05e158ba05c418202124ff067ef018bbc621579646cde8994
Jul 24 18:12:14.035621 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:14.035592    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:12:14.035700 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:14.035652    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:12:14.035700 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:14.035680    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:12:14.247438 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:14.247350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" event={"ID":"a7fef7c5-2e83-4137-ae26-84e22d1b41a9","Type":"ContainerStarted","Data":"45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa"}
Jul 24 18:12:14.247629 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:14.247440    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" event={"ID":"a7fef7c5-2e83-4137-ae26-84e22d1b41a9","Type":"ContainerStarted","Data":"7bb37b05d9e764f05e158ba05c418202124ff067ef018bbc621579646cde8994"}
Jul 24 18:12:14.275655 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:14.275606    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" podStartSLOduration=1.275589468 podStartE2EDuration="1.275589468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:12:14.274138199 +0000 UTC m=+2147.790649236" watchObservedRunningTime="2026-07-24 18:12:14.275589468 +0000 UTC m=+2147.792100501"
Jul 24 18:12:14.900074 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:14.900047    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:15.904920 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:15.904888    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:16.255552 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.255457    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:16.256489 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.256468    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:16.506047 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.505937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:16.642846 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.642806    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:16.774310 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.774231    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"]
Jul 24 18:12:16.774532 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.774519    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerName="istio-proxy"
Jul 24 18:12:16.774591 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.774533    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerName="istio-proxy"
Jul 24 18:12:16.774637 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.774607    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4de454e-5a97-457a-80d9-0bee81717d8c" containerName="istio-proxy"
Jul 24 18:12:16.776923 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.776904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:16.780603 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.780581    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ba2e7e10-serving-cert\""
Jul 24 18:12:16.781697 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.781678    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fvm85\""
Jul 24 18:12:16.781832 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.781815    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:12:16.787283 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.787268    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:12:16.800791 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.800769    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"]
Jul 24 18:12:16.847220 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.847149    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:16.891884 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.891857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af62bb75-8106-46df-9b8b-755f0ce191bb-maas-api-tls\") pod \"maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:16.892017 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.891895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af62bb75-8106-46df-9b8b-755f0ce191bb-openshift-service-ca\") pod \"maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:16.892017 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.891999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbwrc\" (UniqueName: \"kubernetes.io/projected/af62bb75-8106-46df-9b8b-755f0ce191bb-kube-api-access-hbwrc\") pod \"maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:16.993247 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.993217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af62bb75-8106-46df-9b8b-755f0ce191bb-maas-api-tls\") pod \"maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:16.993663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.993257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af62bb75-8106-46df-9b8b-755f0ce191bb-openshift-service-ca\") pod \"maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:16.993663 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:12:16.993363    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-serving-cert: secret "maas-api-e2e-mt-ba2e7e10-serving-cert" not found
Jul 24 18:12:16.993663 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:12:16.993425    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af62bb75-8106-46df-9b8b-755f0ce191bb-maas-api-tls podName:af62bb75-8106-46df-9b8b-755f0ce191bb nodeName:}" failed. No retries permitted until 2026-07-24 18:12:17.493408933 +0000 UTC m=+2151.009919950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af62bb75-8106-46df-9b8b-755f0ce191bb-maas-api-tls") pod "maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw" (UID: "af62bb75-8106-46df-9b8b-755f0ce191bb") : secret "maas-api-e2e-mt-ba2e7e10-serving-cert" not found
Jul 24 18:12:16.993663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.993446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbwrc\" (UniqueName: \"kubernetes.io/projected/af62bb75-8106-46df-9b8b-755f0ce191bb-kube-api-access-hbwrc\") pod \"maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:16.993890 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:16.993873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af62bb75-8106-46df-9b8b-755f0ce191bb-openshift-service-ca\") pod \"maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:17.003701 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.003681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbwrc\" (UniqueName: \"kubernetes.io/projected/af62bb75-8106-46df-9b8b-755f0ce191bb-kube-api-access-hbwrc\") pod \"maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:17.224315 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.224280    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:17.455099 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.455054    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:17.497008 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.496938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af62bb75-8106-46df-9b8b-755f0ce191bb-maas-api-tls\") pod \"maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:17.499188 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.499170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af62bb75-8106-46df-9b8b-755f0ce191bb-maas-api-tls\") pod \"maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:17.560807 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.560778    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:17.653667 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.653636    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"]
Jul 24 18:12:17.656918 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.656901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"
Jul 24 18:12:17.660760 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.660740    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ba2e7e10-dockercfg-dmgq4\""
Jul 24 18:12:17.661102 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.661088    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ba2e7e10\""
Jul 24 18:12:17.666116 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.666095    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"]
Jul 24 18:12:17.687111 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.687091    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:17.699327 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.699301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffgt\" (UniqueName: \"kubernetes.io/projected/e10391b3-32ad-40ea-aa7e-927da085d359-kube-api-access-2ffgt\") pod \"payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4\" (UID: \"e10391b3-32ad-40ea-aa7e-927da085d359\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"
Jul 24 18:12:17.699599 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.699559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10391b3-32ad-40ea-aa7e-927da085d359-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4\" (UID: \"e10391b3-32ad-40ea-aa7e-927da085d359\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"
Jul 24 18:12:17.719934 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.719904    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"]
Jul 24 18:12:17.723236 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.723219    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"
Jul 24 18:12:17.734671 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.734646    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"]
Jul 24 18:12:17.800750 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.800710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffgt\" (UniqueName: \"kubernetes.io/projected/e10391b3-32ad-40ea-aa7e-927da085d359-kube-api-access-2ffgt\") pod \"payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4\" (UID: \"e10391b3-32ad-40ea-aa7e-927da085d359\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"
Jul 24 18:12:17.800903 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.800778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptnbj\" (UniqueName: \"kubernetes.io/projected/c45ffe05-0399-4702-a5e1-eebf186607ff-kube-api-access-ptnbj\") pod \"payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw\" (UID: \"c45ffe05-0399-4702-a5e1-eebf186607ff\") " pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"
Jul 24 18:12:17.800903 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.800856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10391b3-32ad-40ea-aa7e-927da085d359-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4\" (UID: \"e10391b3-32ad-40ea-aa7e-927da085d359\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"
Jul 24 18:12:17.800903 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.800890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c45ffe05-0399-4702-a5e1-eebf186607ff-plugins-config-volume\") pod \"payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw\" (UID: \"c45ffe05-0399-4702-a5e1-eebf186607ff\") " pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"
Jul 24 18:12:17.801538 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.801514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10391b3-32ad-40ea-aa7e-927da085d359-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4\" (UID: \"e10391b3-32ad-40ea-aa7e-927da085d359\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"
Jul 24 18:12:17.810151 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.810130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffgt\" (UniqueName: \"kubernetes.io/projected/e10391b3-32ad-40ea-aa7e-927da085d359-kube-api-access-2ffgt\") pod \"payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4\" (UID: \"e10391b3-32ad-40ea-aa7e-927da085d359\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"
Jul 24 18:12:17.831243 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.831221    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"]
Jul 24 18:12:17.833994 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:12:17.833970    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf62bb75_8106_46df_9b8b_755f0ce191bb.slice/crio-7d2b5dd8ca9da33456ac9e02fb40ca5bcc53a965daa3eaaf0f40d0e2f8503001 WatchSource:0}: Error finding container 7d2b5dd8ca9da33456ac9e02fb40ca5bcc53a965daa3eaaf0f40d0e2f8503001: Status 404 returned error can't find the container with id 7d2b5dd8ca9da33456ac9e02fb40ca5bcc53a965daa3eaaf0f40d0e2f8503001
Jul 24 18:12:17.902260 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.902234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptnbj\" (UniqueName: \"kubernetes.io/projected/c45ffe05-0399-4702-a5e1-eebf186607ff-kube-api-access-ptnbj\") pod \"payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw\" (UID: \"c45ffe05-0399-4702-a5e1-eebf186607ff\") " pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"
Jul 24 18:12:17.902392 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.902289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c45ffe05-0399-4702-a5e1-eebf186607ff-plugins-config-volume\") pod \"payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw\" (UID: \"c45ffe05-0399-4702-a5e1-eebf186607ff\") " pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"
Jul 24 18:12:17.902943 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.902923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c45ffe05-0399-4702-a5e1-eebf186607ff-plugins-config-volume\") pod \"payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw\" (UID: \"c45ffe05-0399-4702-a5e1-eebf186607ff\") " pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"
Jul 24 18:12:17.915922 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.915894    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptnbj\" (UniqueName: \"kubernetes.io/projected/c45ffe05-0399-4702-a5e1-eebf186607ff-kube-api-access-ptnbj\") pod \"payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw\" (UID: \"c45ffe05-0399-4702-a5e1-eebf186607ff\") " pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"
Jul 24 18:12:17.967610 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:17.967554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"
Jul 24 18:12:18.039201 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:18.039119    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"
Jul 24 18:12:18.150107 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:18.150079    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"]
Jul 24 18:12:18.152759 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:12:18.152730    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10391b3_32ad_40ea_aa7e_927da085d359.slice/crio-9ff68bbd61d3e93c411f5e1872f2445ae70e9cf9d45cc0fa9602b28aaf374b3b WatchSource:0}: Error finding container 9ff68bbd61d3e93c411f5e1872f2445ae70e9cf9d45cc0fa9602b28aaf374b3b: Status 404 returned error can't find the container with id 9ff68bbd61d3e93c411f5e1872f2445ae70e9cf9d45cc0fa9602b28aaf374b3b
Jul 24 18:12:18.195862 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:18.195834    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"]
Jul 24 18:12:18.198829 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:12:18.198802    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45ffe05_0399_4702_a5e1_eebf186607ff.slice/crio-3914d71759b6b0f27230f5db977175bd22615f53b1db354a1f1e85e64a8ad1a7 WatchSource:0}: Error finding container 3914d71759b6b0f27230f5db977175bd22615f53b1db354a1f1e85e64a8ad1a7: Status 404 returned error can't find the container with id 3914d71759b6b0f27230f5db977175bd22615f53b1db354a1f1e85e64a8ad1a7
Jul 24 18:12:18.265639 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:18.265601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw" event={"ID":"c45ffe05-0399-4702-a5e1-eebf186607ff","Type":"ContainerStarted","Data":"9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2"}
Jul 24 18:12:18.265822 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:18.265665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw" event={"ID":"c45ffe05-0399-4702-a5e1-eebf186607ff","Type":"ContainerStarted","Data":"3914d71759b6b0f27230f5db977175bd22615f53b1db354a1f1e85e64a8ad1a7"}
Jul 24 18:12:18.265822 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:18.265694    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"
Jul 24 18:12:18.267314 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:18.267287    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4" event={"ID":"e10391b3-32ad-40ea-aa7e-927da085d359","Type":"ContainerStarted","Data":"9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8"}
Jul 24 18:12:18.267435 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:18.267334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4" event={"ID":"e10391b3-32ad-40ea-aa7e-927da085d359","Type":"ContainerStarted","Data":"9ff68bbd61d3e93c411f5e1872f2445ae70e9cf9d45cc0fa9602b28aaf374b3b"}
Jul 24 18:12:18.267435 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:18.267376    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"
Jul 24 18:12:18.269185 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:18.269160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw" event={"ID":"af62bb75-8106-46df-9b8b-755f0ce191bb","Type":"ContainerStarted","Data":"7d2b5dd8ca9da33456ac9e02fb40ca5bcc53a965daa3eaaf0f40d0e2f8503001"}
Jul 24 18:12:18.289503 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:18.289415    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw" podStartSLOduration=1.289401521 podStartE2EDuration="1.289401521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:12:18.286937262 +0000 UTC m=+2151.803448298" watchObservedRunningTime="2026-07-24 18:12:18.289401521 +0000 UTC m=+2151.805912556"
Jul 24 18:12:18.309851 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:18.309810    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4" podStartSLOduration=1.309795366 podStartE2EDuration="1.309795366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:12:18.308641472 +0000 UTC m=+2151.825152509" watchObservedRunningTime="2026-07-24 18:12:18.309795366 +0000 UTC m=+2151.826306401"
Jul 24 18:12:19.275789 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:19.275697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw" event={"ID":"af62bb75-8106-46df-9b8b-755f0ce191bb","Type":"ContainerStarted","Data":"0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0"}
Jul 24 18:12:19.276209 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:19.276089    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:19.301692 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:19.301647    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw" podStartSLOduration=2.792310442 podStartE2EDuration="3.301629112s" totalImagesPullingTime="509.31867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:12:19.300907979 +0000 UTC m=+2152.817419017" watchObservedRunningTime="2026-07-24 18:12:19.301629112 +0000 UTC m=+2152.818140150"
Jul 24 18:12:21.010683 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:21.010642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:21.649489 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:21.649462    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:21.759242 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:21.759213    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:22.114639 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:22.114601    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:26.291283 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:26.291254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:27.908087 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:27.908048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:28.015429 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:28.015397    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:29.277916 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:29.277883    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"
Jul 24 18:12:29.278410 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:29.278093    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"
Jul 24 18:12:30.335060 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:30.335024    2581 generic.go:356] "Generic (PLEG): container finished" podID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerID="45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa" exitCode=0
Jul 24 18:12:30.335468 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:30.335074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" event={"ID":"a7fef7c5-2e83-4137-ae26-84e22d1b41a9","Type":"ContainerDied","Data":"45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa"}
Jul 24 18:12:30.335468 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:30.335437    2581 scope.go:117] "RemoveContainer" containerID="45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa"
Jul 24 18:12:30.336420 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:30.336394    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:12:30.336483 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:30.336461    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:12:30.336526 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:30.336491    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:12:31.255511 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:31.255480    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:31.341470 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:31.341434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" event={"ID":"a7fef7c5-2e83-4137-ae26-84e22d1b41a9","Type":"ContainerStarted","Data":"c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd"}
Jul 24 18:12:31.899652 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:31.899618    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:31.900844 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:31.900816    2581 patch_prober.go:28] interesting pod/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Jul 24 18:12:31.900974 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:31.900865    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Jul 24 18:12:32.900674 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:32.900638    2581 patch_prober.go:28] interesting pod/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Jul 24 18:12:32.901067 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:32.900705    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Jul 24 18:12:33.899978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:33.899942    2581 patch_prober.go:28] interesting pod/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Jul 24 18:12:33.900151 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:33.900000    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Jul 24 18:12:34.900551 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:34.900512    2581 patch_prober.go:28] interesting pod/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Jul 24 18:12:34.900961 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:34.900592    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Jul 24 18:12:35.899827 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:35.899783    2581 patch_prober.go:28] interesting pod/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Jul 24 18:12:35.900014 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:35.899843    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Jul 24 18:12:36.904396 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:36.904361    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:36.904882 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:36.904756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:36.905496 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:36.905476    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:38.730553 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:38.730519    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"]
Jul 24 18:12:38.730968 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:38.730768    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw" podUID="af62bb75-8106-46df-9b8b-755f0ce191bb" containerName="maas-api" containerID="cri-o://0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0" gracePeriod=30
Jul 24 18:12:38.765805 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:38.765780    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:38.787431 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:38.787403    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"]
Jul 24 18:12:38.787675 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:38.787627    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw" podUID="c45ffe05-0399-4702-a5e1-eebf186607ff" containerName="payload-processing" containerID="cri-o://9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2" gracePeriod=30
Jul 24 18:12:38.796354 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:38.796327    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"]
Jul 24 18:12:38.796611 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:38.796562    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4" podUID="e10391b3-32ad-40ea-aa7e-927da085d359" containerName="payload-pre-processing" containerID="cri-o://9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8" gracePeriod=30
Jul 24 18:12:38.830536 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:38.830509    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:38.892338 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:38.892309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:39.048346 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.048322    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:39.075341 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.075317    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"
Jul 24 18:12:39.084492 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.084470    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"
Jul 24 18:12:39.202992 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.202949    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ffgt\" (UniqueName: \"kubernetes.io/projected/e10391b3-32ad-40ea-aa7e-927da085d359-kube-api-access-2ffgt\") pod \"e10391b3-32ad-40ea-aa7e-927da085d359\" (UID: \"e10391b3-32ad-40ea-aa7e-927da085d359\") "
Jul 24 18:12:39.203228 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.203008    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbwrc\" (UniqueName: \"kubernetes.io/projected/af62bb75-8106-46df-9b8b-755f0ce191bb-kube-api-access-hbwrc\") pod \"af62bb75-8106-46df-9b8b-755f0ce191bb\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") "
Jul 24 18:12:39.203228 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.203028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10391b3-32ad-40ea-aa7e-927da085d359-plugins-config-volume\") pod \"e10391b3-32ad-40ea-aa7e-927da085d359\" (UID: \"e10391b3-32ad-40ea-aa7e-927da085d359\") "
Jul 24 18:12:39.203228 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.203142    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c45ffe05-0399-4702-a5e1-eebf186607ff-plugins-config-volume\") pod \"c45ffe05-0399-4702-a5e1-eebf186607ff\" (UID: \"c45ffe05-0399-4702-a5e1-eebf186607ff\") "
Jul 24 18:12:39.203228 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.203200    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af62bb75-8106-46df-9b8b-755f0ce191bb-maas-api-tls\") pod \"af62bb75-8106-46df-9b8b-755f0ce191bb\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") "
Jul 24 18:12:39.203228 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.203227    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af62bb75-8106-46df-9b8b-755f0ce191bb-openshift-service-ca\") pod \"af62bb75-8106-46df-9b8b-755f0ce191bb\" (UID: \"af62bb75-8106-46df-9b8b-755f0ce191bb\") "
Jul 24 18:12:39.203500 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.203274    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptnbj\" (UniqueName: \"kubernetes.io/projected/c45ffe05-0399-4702-a5e1-eebf186607ff-kube-api-access-ptnbj\") pod \"c45ffe05-0399-4702-a5e1-eebf186607ff\" (UID: \"c45ffe05-0399-4702-a5e1-eebf186607ff\") "
Jul 24 18:12:39.203500 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.203392    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10391b3-32ad-40ea-aa7e-927da085d359-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e10391b3-32ad-40ea-aa7e-927da085d359" (UID: "e10391b3-32ad-40ea-aa7e-927da085d359"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:12:39.203643 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.203539    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10391b3-32ad-40ea-aa7e-927da085d359-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:39.203643 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.203604    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45ffe05-0399-4702-a5e1-eebf186607ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c45ffe05-0399-4702-a5e1-eebf186607ff" (UID: "c45ffe05-0399-4702-a5e1-eebf186607ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:12:39.203749 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.203682    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af62bb75-8106-46df-9b8b-755f0ce191bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af62bb75-8106-46df-9b8b-755f0ce191bb" (UID: "af62bb75-8106-46df-9b8b-755f0ce191bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:12:39.205237 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.205212    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af62bb75-8106-46df-9b8b-755f0ce191bb-kube-api-access-hbwrc" (OuterVolumeSpecName: "kube-api-access-hbwrc") pod "af62bb75-8106-46df-9b8b-755f0ce191bb" (UID: "af62bb75-8106-46df-9b8b-755f0ce191bb"). InnerVolumeSpecName "kube-api-access-hbwrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:12:39.205237 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.205225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10391b3-32ad-40ea-aa7e-927da085d359-kube-api-access-2ffgt" (OuterVolumeSpecName: "kube-api-access-2ffgt") pod "e10391b3-32ad-40ea-aa7e-927da085d359" (UID: "e10391b3-32ad-40ea-aa7e-927da085d359"). InnerVolumeSpecName "kube-api-access-2ffgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:12:39.205433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.205225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af62bb75-8106-46df-9b8b-755f0ce191bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af62bb75-8106-46df-9b8b-755f0ce191bb" (UID: "af62bb75-8106-46df-9b8b-755f0ce191bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:12:39.205433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.205310    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45ffe05-0399-4702-a5e1-eebf186607ff-kube-api-access-ptnbj" (OuterVolumeSpecName: "kube-api-access-ptnbj") pod "c45ffe05-0399-4702-a5e1-eebf186607ff" (UID: "c45ffe05-0399-4702-a5e1-eebf186607ff"). InnerVolumeSpecName "kube-api-access-ptnbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:12:39.304737 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.304658    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c45ffe05-0399-4702-a5e1-eebf186607ff-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:39.304737 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.304683    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af62bb75-8106-46df-9b8b-755f0ce191bb-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:39.304737 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.304693    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af62bb75-8106-46df-9b8b-755f0ce191bb-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:39.304737 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.304702    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptnbj\" (UniqueName: \"kubernetes.io/projected/c45ffe05-0399-4702-a5e1-eebf186607ff-kube-api-access-ptnbj\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:39.304737 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.304712    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ffgt\" (UniqueName: \"kubernetes.io/projected/e10391b3-32ad-40ea-aa7e-927da085d359-kube-api-access-2ffgt\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:39.304737 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.304724    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbwrc\" (UniqueName: \"kubernetes.io/projected/af62bb75-8106-46df-9b8b-755f0ce191bb-kube-api-access-hbwrc\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:39.397773 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.397739    2581 generic.go:356] "Generic (PLEG): container finished" podID="e10391b3-32ad-40ea-aa7e-927da085d359" containerID="9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8" exitCode=0
Jul 24 18:12:39.397959 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.397782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4" event={"ID":"e10391b3-32ad-40ea-aa7e-927da085d359","Type":"ContainerDied","Data":"9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8"}
Jul 24 18:12:39.397959 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.397793    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"
Jul 24 18:12:39.397959 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.397874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4" event={"ID":"e10391b3-32ad-40ea-aa7e-927da085d359","Type":"ContainerDied","Data":"9ff68bbd61d3e93c411f5e1872f2445ae70e9cf9d45cc0fa9602b28aaf374b3b"}
Jul 24 18:12:39.397959 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.397907    2581 scope.go:117] "RemoveContainer" containerID="9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8"
Jul 24 18:12:39.399348 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.399325    2581 generic.go:356] "Generic (PLEG): container finished" podID="af62bb75-8106-46df-9b8b-755f0ce191bb" containerID="0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0" exitCode=0
Jul 24 18:12:39.399444 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.399387    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"
Jul 24 18:12:39.399444 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.399413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw" event={"ID":"af62bb75-8106-46df-9b8b-755f0ce191bb","Type":"ContainerDied","Data":"0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0"}
Jul 24 18:12:39.399547 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.399462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw" event={"ID":"af62bb75-8106-46df-9b8b-755f0ce191bb","Type":"ContainerDied","Data":"7d2b5dd8ca9da33456ac9e02fb40ca5bcc53a965daa3eaaf0f40d0e2f8503001"}
Jul 24 18:12:39.400960 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.400936    2581 generic.go:356] "Generic (PLEG): container finished" podID="c45ffe05-0399-4702-a5e1-eebf186607ff" containerID="9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2" exitCode=0
Jul 24 18:12:39.401081 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.400963    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw" event={"ID":"c45ffe05-0399-4702-a5e1-eebf186607ff","Type":"ContainerDied","Data":"9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2"}
Jul 24 18:12:39.401081 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.400995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw" event={"ID":"c45ffe05-0399-4702-a5e1-eebf186607ff","Type":"ContainerDied","Data":"3914d71759b6b0f27230f5db977175bd22615f53b1db354a1f1e85e64a8ad1a7"}
Jul 24 18:12:39.401081 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.401007    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"
Jul 24 18:12:39.406586 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.406549    2581 scope.go:117] "RemoveContainer" containerID="9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8"
Jul 24 18:12:39.406846 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:12:39.406816    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8\": container with ID starting with 9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8 not found: ID does not exist" containerID="9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8"
Jul 24 18:12:39.406966 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.406854    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8"} err="failed to get container status \"9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8\": rpc error: code = NotFound desc = could not find container \"9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8\": container with ID starting with 9f53f54f7dd218d42411db01fbbe7d44d4e467868e93611ed526147560e24db8 not found: ID does not exist"
Jul 24 18:12:39.406966 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.406877    2581 scope.go:117] "RemoveContainer" containerID="0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0"
Jul 24 18:12:39.415909 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.415893    2581 scope.go:117] "RemoveContainer" containerID="0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0"
Jul 24 18:12:39.416145 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:12:39.416127    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0\": container with ID starting with 0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0 not found: ID does not exist" containerID="0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0"
Jul 24 18:12:39.416195 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.416152    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0"} err="failed to get container status \"0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0\": rpc error: code = NotFound desc = could not find container \"0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0\": container with ID starting with 0fd253a88bc7d1a0758a4ce9a7445a727ff38ee26667b4a008c8ee3559c5e3c0 not found: ID does not exist"
Jul 24 18:12:39.416195 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.416168    2581 scope.go:117] "RemoveContainer" containerID="9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2"
Jul 24 18:12:39.423138 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.423121    2581 scope.go:117] "RemoveContainer" containerID="9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2"
Jul 24 18:12:39.423383 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:12:39.423365    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2\": container with ID starting with 9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2 not found: ID does not exist" containerID="9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2"
Jul 24 18:12:39.423462 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.423390    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2"} err="failed to get container status \"9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2\": rpc error: code = NotFound desc = could not find container \"9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2\": container with ID starting with 9042b784e443454d32d50dae9dd603b5894c1d9e3f938fcc8b48e28adefa18c2 not found: ID does not exist"
Jul 24 18:12:39.424765 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.424688    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"]
Jul 24 18:12:39.426187 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.426155    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba2e7e10-67d57bf98d-nr8mw"]
Jul 24 18:12:39.434452 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.434430    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"]
Jul 24 18:12:39.436290 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.436269    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba2e7e10-7b774967b4-jb8s4"]
Jul 24 18:12:39.445590 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.445553    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"]
Jul 24 18:12:39.447731 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:39.447713    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba2e7e10-84d8f8c9bb-ptpcw"]
Jul 24 18:12:40.957137 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:40.957095    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af62bb75-8106-46df-9b8b-755f0ce191bb" path="/var/lib/kubelet/pods/af62bb75-8106-46df-9b8b-755f0ce191bb/volumes"
Jul 24 18:12:40.957653 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:40.957603    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c45ffe05-0399-4702-a5e1-eebf186607ff" path="/var/lib/kubelet/pods/c45ffe05-0399-4702-a5e1-eebf186607ff/volumes"
Jul 24 18:12:40.957985 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:40.957964    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e10391b3-32ad-40ea-aa7e-927da085d359" path="/var/lib/kubelet/pods/e10391b3-32ad-40ea-aa7e-927da085d359/volumes"
Jul 24 18:12:43.391543 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:43.391506    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:43.798385 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:43.798291    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:49.729383 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:49.729350    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:49.829733 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:49.829695    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:50.005718 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:50.005636    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"]
Jul 24 18:12:50.005903 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:50.005878    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy" containerID="cri-o://c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd" gracePeriod=30
Jul 24 18:12:50.030233 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:50.030199    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:50.260868 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:50.260788    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:50.396733 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:50.396693    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:50.498697 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:50.498653    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:50.629921 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:50.629892    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:51.907287 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:51.907248    2581 patch_prober.go:28] interesting pod/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 18:12:51.907684 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:51.907317    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 18:12:55.250663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.250643    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:55.322308 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322283    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istiod-ca-cert\") pod \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") "
Jul 24 18:12:55.322442 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322319    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-podinfo\") pod \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") "
Jul 24 18:12:55.322442 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322353    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-token\") pod \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") "
Jul 24 18:12:55.322442 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322388    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-envoy\") pod \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") "
Jul 24 18:12:55.322442 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322411    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whzct\" (UniqueName: \"kubernetes.io/projected/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-kube-api-access-whzct\") pod \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") "
Jul 24 18:12:55.322442 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322439    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-workload-socket\") pod \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") "
Jul 24 18:12:55.322720 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322464    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-workload-certs\") pod \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") "
Jul 24 18:12:55.322720 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322493    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-data\") pod \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") "
Jul 24 18:12:55.322720 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322538    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-credential-socket\") pod \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\" (UID: \"a7fef7c5-2e83-4137-ae26-84e22d1b41a9\") "
Jul 24 18:12:55.322720 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322711    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7fef7c5-2e83-4137-ae26-84e22d1b41a9" (UID: "a7fef7c5-2e83-4137-ae26-84e22d1b41a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:12:55.322920 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322794    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:55.322920 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322811    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7fef7c5-2e83-4137-ae26-84e22d1b41a9" (UID: "a7fef7c5-2e83-4137-ae26-84e22d1b41a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:12:55.323036 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.322984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7fef7c5-2e83-4137-ae26-84e22d1b41a9" (UID: "a7fef7c5-2e83-4137-ae26-84e22d1b41a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:12:55.323036 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.323010    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7fef7c5-2e83-4137-ae26-84e22d1b41a9" (UID: "a7fef7c5-2e83-4137-ae26-84e22d1b41a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:12:55.323764 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.323739    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7fef7c5-2e83-4137-ae26-84e22d1b41a9" (UID: "a7fef7c5-2e83-4137-ae26-84e22d1b41a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:12:55.324643 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.324612    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7fef7c5-2e83-4137-ae26-84e22d1b41a9" (UID: "a7fef7c5-2e83-4137-ae26-84e22d1b41a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:12:55.324786 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.324765    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-kube-api-access-whzct" (OuterVolumeSpecName: "kube-api-access-whzct") pod "a7fef7c5-2e83-4137-ae26-84e22d1b41a9" (UID: "a7fef7c5-2e83-4137-ae26-84e22d1b41a9"). InnerVolumeSpecName "kube-api-access-whzct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:12:55.324948 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.324927    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7fef7c5-2e83-4137-ae26-84e22d1b41a9" (UID: "a7fef7c5-2e83-4137-ae26-84e22d1b41a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 18:12:55.325048 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.324928    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7fef7c5-2e83-4137-ae26-84e22d1b41a9" (UID: "a7fef7c5-2e83-4137-ae26-84e22d1b41a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:12:55.423195 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.423150    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:55.423195 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.423189    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whzct\" (UniqueName: \"kubernetes.io/projected/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-kube-api-access-whzct\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:55.423195 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.423201    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:55.423195 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.423209    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:55.423444 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.423218    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:55.423444 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.423226    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:55.423444 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.423233    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:55.423444 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.423241    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7fef7c5-2e83-4137-ae26-84e22d1b41a9-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:12:55.472151 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.472116    2581 generic.go:356] "Generic (PLEG): container finished" podID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerID="c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd" exitCode=0
Jul 24 18:12:55.472292 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.472160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" event={"ID":"a7fef7c5-2e83-4137-ae26-84e22d1b41a9","Type":"ContainerDied","Data":"c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd"}
Jul 24 18:12:55.472292 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.472219    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp" event={"ID":"a7fef7c5-2e83-4137-ae26-84e22d1b41a9","Type":"ContainerDied","Data":"7bb37b05d9e764f05e158ba05c418202124ff067ef018bbc621579646cde8994"}
Jul 24 18:12:55.472292 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.472172    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"
Jul 24 18:12:55.472292 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.472244    2581 scope.go:117] "RemoveContainer" containerID="c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd"
Jul 24 18:12:55.480793 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.480771    2581 scope.go:117] "RemoveContainer" containerID="45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa"
Jul 24 18:12:55.488157 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.488138    2581 scope.go:117] "RemoveContainer" containerID="c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd"
Jul 24 18:12:55.488415 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:12:55.488395    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd\": container with ID starting with c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd not found: ID does not exist" containerID="c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd"
Jul 24 18:12:55.488479 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.488423    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd"} err="failed to get container status \"c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd\": rpc error: code = NotFound desc = could not find container \"c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd\": container with ID starting with c547b1038f3a130b5d299e4bda715539e03a43efb29adf058812b3a2b1a625dd not found: ID does not exist"
Jul 24 18:12:55.488479 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.488440    2581 scope.go:117] "RemoveContainer" containerID="45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa"
Jul 24 18:12:55.488744 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:12:55.488725    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa\": container with ID starting with 45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa not found: ID does not exist" containerID="45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa"
Jul 24 18:12:55.488794 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.488750    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa"} err="failed to get container status \"45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa\": rpc error: code = NotFound desc = could not find container \"45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa\": container with ID starting with 45244f7ba55735097ee1308672443efd29e1ee6c699a93133e462e1ca15c2caa not found: ID does not exist"
Jul 24 18:12:55.493351 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.493323    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"]
Jul 24 18:12:55.497294 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:55.497272    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ba2e7e10-openshift-default-5686cf6c76-wk5gp"]
Jul 24 18:12:56.381177 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:56.381139    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:56.453818 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:56.453788    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:56.539823 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:56.539789    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:56.639069 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:56.638991    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:56.764193 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:56.764166    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:12:56.957791 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:12:56.957716    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" path="/var/lib/kubelet/pods/a7fef7c5-2e83-4137-ae26-84e22d1b41a9/volumes"
Jul 24 18:13:02.480412 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:02.480381    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:02.584117 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:02.584075    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:09.396192 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.396161    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:09.678932 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.678848    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"]
Jul 24 18:13:09.679128 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679115    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af62bb75-8106-46df-9b8b-755f0ce191bb" containerName="maas-api"
Jul 24 18:13:09.679128 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679128    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="af62bb75-8106-46df-9b8b-755f0ce191bb" containerName="maas-api"
Jul 24 18:13:09.679225 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679137    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy"
Jul 24 18:13:09.679225 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679144    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy"
Jul 24 18:13:09.679225 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679152    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10391b3-32ad-40ea-aa7e-927da085d359" containerName="payload-pre-processing"
Jul 24 18:13:09.679225 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679158    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10391b3-32ad-40ea-aa7e-927da085d359" containerName="payload-pre-processing"
Jul 24 18:13:09.679225 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679163    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy"
Jul 24 18:13:09.679225 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679169    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy"
Jul 24 18:13:09.679225 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679180    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c45ffe05-0399-4702-a5e1-eebf186607ff" containerName="payload-processing"
Jul 24 18:13:09.679225 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679185    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c45ffe05-0399-4702-a5e1-eebf186607ff" containerName="payload-processing"
Jul 24 18:13:09.679509 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679234    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy"
Jul 24 18:13:09.679509 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679243    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="af62bb75-8106-46df-9b8b-755f0ce191bb" containerName="maas-api"
Jul 24 18:13:09.679509 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679250    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10391b3-32ad-40ea-aa7e-927da085d359" containerName="payload-pre-processing"
Jul 24 18:13:09.679509 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.679256    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c45ffe05-0399-4702-a5e1-eebf186607ff" containerName="payload-processing"
Jul 24 18:13:09.682018 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.682000    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.684541 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.684525    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a046f5ba-openshift-default-dockercfg-4vfkv\""
Jul 24 18:13:09.693506 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.693480    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"]
Jul 24 18:13:09.708644 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.708621    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:09.726229 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.726193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-credential-socket\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.726333 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.726267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-token\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.726333 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.726309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-workload-socket\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.726333 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.726326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-workload-certs\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.726488 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.726340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-envoy\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.726488 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.726355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-data\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.726488 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.726434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzgx2\" (UniqueName: \"kubernetes.io/projected/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-kube-api-access-lzgx2\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.726488 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.726474    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-podinfo\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.726666 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.726494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istiod-ca-cert\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.827939 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.827899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-credential-socket\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.828109 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.827953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-token\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.828109 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.827990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-workload-socket\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.828109 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.828013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-workload-certs\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.828109 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.828034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-envoy\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.828333 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.828168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-data\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.828333 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.828226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzgx2\" (UniqueName: \"kubernetes.io/projected/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-kube-api-access-lzgx2\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.828333 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.828268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-podinfo\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.828333 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.828293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istiod-ca-cert\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.828333 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.828307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-credential-socket\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.828613 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.828331    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-workload-certs\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.828613 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.828416    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-workload-socket\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.828613 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.828547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-data\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.829003 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.828982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istiod-ca-cert\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.830464 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.830441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-envoy\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.830699 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.830682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-podinfo\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.835697 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.835673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-token\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.835960 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.835942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzgx2\" (UniqueName: \"kubernetes.io/projected/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-kube-api-access-lzgx2\") pod \"e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") " pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:09.936328 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.936248    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:09.991808 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:09.991779    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:10.122003 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:10.121978    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"]
Jul 24 18:13:10.124472 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:13:10.124444    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de7e79f_015a_4f9e_bdbd_b19f5f3afd41.slice/crio-b8f84e6ae26848941e561bd3bb66700aba1778cc58804d6e16e2d34273715562 WatchSource:0}: Error finding container b8f84e6ae26848941e561bd3bb66700aba1778cc58804d6e16e2d34273715562: Status 404 returned error can't find the container with id b8f84e6ae26848941e561bd3bb66700aba1778cc58804d6e16e2d34273715562
Jul 24 18:13:10.126780 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:10.126743    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:13:10.126851 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:10.126823    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:13:10.126897 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:10.126862    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:13:10.537693 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:10.537659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6" event={"ID":"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41","Type":"ContainerStarted","Data":"e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f"}
Jul 24 18:13:10.538057 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:10.537706    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6" event={"ID":"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41","Type":"ContainerStarted","Data":"b8f84e6ae26848941e561bd3bb66700aba1778cc58804d6e16e2d34273715562"}
Jul 24 18:13:10.563022 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:10.562974    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6" podStartSLOduration=1.562956011 podStartE2EDuration="1.562956011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:13:10.561041378 +0000 UTC m=+2204.077552418" watchObservedRunningTime="2026-07-24 18:13:10.562956011 +0000 UTC m=+2204.079467046"
Jul 24 18:13:10.992355 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:10.992328    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:11.997102 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:11.997067    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:12.540815 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.540785    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:12.546493 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.546472    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:12.547492 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.547473    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:12.617506 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.617466    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"]
Jul 24 18:13:12.617954 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.617934    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7fef7c5-2e83-4137-ae26-84e22d1b41a9" containerName="istio-proxy"
Jul 24 18:13:12.625197 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.625173    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:12.628917 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.628899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:12.631288 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.631267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fvm85\""
Jul 24 18:13:12.631414 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.631318    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:13:12.631414 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.631341    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:13:12.631782 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.631760    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a046f5ba-serving-cert\""
Jul 24 18:13:12.636125 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.636104    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"]
Jul 24 18:13:12.649857 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.649833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-openshift-service-ca\") pod \"maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:12.649979 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.649921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-maas-api-tls\") pod \"maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:12.650063 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.650041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5n57\" (UniqueName: \"kubernetes.io/projected/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-kube-api-access-r5n57\") pod \"maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:12.695868 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.695845    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:12.750682 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.750642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-openshift-service-ca\") pod \"maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:12.750860 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.750729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-maas-api-tls\") pod \"maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:12.750860 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.750773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5n57\" (UniqueName: \"kubernetes.io/projected/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-kube-api-access-r5n57\") pod \"maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:12.750956 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:13:12.750856    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-serving-cert: secret "maas-api-e2e-mt-a046f5ba-serving-cert" not found
Jul 24 18:13:12.750956 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:13:12.750938    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-maas-api-tls podName:8d3316f5-b443-4df4-bdaa-bcd3e8f438b9 nodeName:}" failed. No retries permitted until 2026-07-24 18:13:13.250917333 +0000 UTC m=+2206.767428349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-maas-api-tls") pod "maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz" (UID: "8d3316f5-b443-4df4-bdaa-bcd3e8f438b9") : secret "maas-api-e2e-mt-a046f5ba-serving-cert" not found
Jul 24 18:13:12.751366 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.751348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-openshift-service-ca\") pod \"maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:12.759233 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:12.759211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5n57\" (UniqueName: \"kubernetes.io/projected/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-kube-api-access-r5n57\") pod \"maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:13.113508 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.113475    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:13.254852 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.254815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-maas-api-tls\") pod \"maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:13.257232 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.257208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-maas-api-tls\") pod \"maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:13.371325 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.371236    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:13.461039 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.461006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:13.470585 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.470541    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"]
Jul 24 18:13:13.476437 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.475707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"
Jul 24 18:13:13.479314 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.479284    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a046f5ba\""
Jul 24 18:13:13.479439 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.479322    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a046f5ba-dockercfg-pf2hz\""
Jul 24 18:13:13.486292 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.486267    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"]
Jul 24 18:13:13.497048 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.497024    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"]
Jul 24 18:13:13.500464 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.500449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"
Jul 24 18:13:13.510674 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.510656    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"]
Jul 24 18:13:13.541184 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.541153    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:13.557701 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.557670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89615b27-98e1-41c7-a530-2c2ab0abee03-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf\" (UID: \"89615b27-98e1-41c7-a530-2c2ab0abee03\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"
Jul 24 18:13:13.557807 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.557760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7dg\" (UniqueName: \"kubernetes.io/projected/89615b27-98e1-41c7-a530-2c2ab0abee03-kube-api-access-nw7dg\") pod \"payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf\" (UID: \"89615b27-98e1-41c7-a530-2c2ab0abee03\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"
Jul 24 18:13:13.557876 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.557830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822d7228-6aa5-49d6-b26b-e1caa50f1212-plugins-config-volume\") pod \"payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj\" (UID: \"822d7228-6aa5-49d6-b26b-e1caa50f1212\") " pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"
Jul 24 18:13:13.557945 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.557924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxbc2\" (UniqueName: \"kubernetes.io/projected/822d7228-6aa5-49d6-b26b-e1caa50f1212-kube-api-access-nxbc2\") pod \"payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj\" (UID: \"822d7228-6aa5-49d6-b26b-e1caa50f1212\") " pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"
Jul 24 18:13:13.659335 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.659303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89615b27-98e1-41c7-a530-2c2ab0abee03-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf\" (UID: \"89615b27-98e1-41c7-a530-2c2ab0abee03\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"
Jul 24 18:13:13.659489 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.659356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7dg\" (UniqueName: \"kubernetes.io/projected/89615b27-98e1-41c7-a530-2c2ab0abee03-kube-api-access-nw7dg\") pod \"payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf\" (UID: \"89615b27-98e1-41c7-a530-2c2ab0abee03\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"
Jul 24 18:13:13.659489 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.659395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822d7228-6aa5-49d6-b26b-e1caa50f1212-plugins-config-volume\") pod \"payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj\" (UID: \"822d7228-6aa5-49d6-b26b-e1caa50f1212\") " pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"
Jul 24 18:13:13.659630 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.659487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxbc2\" (UniqueName: \"kubernetes.io/projected/822d7228-6aa5-49d6-b26b-e1caa50f1212-kube-api-access-nxbc2\") pod \"payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj\" (UID: \"822d7228-6aa5-49d6-b26b-e1caa50f1212\") " pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"
Jul 24 18:13:13.660206 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.660172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89615b27-98e1-41c7-a530-2c2ab0abee03-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf\" (UID: \"89615b27-98e1-41c7-a530-2c2ab0abee03\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"
Jul 24 18:13:13.660324 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.660272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822d7228-6aa5-49d6-b26b-e1caa50f1212-plugins-config-volume\") pod \"payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj\" (UID: \"822d7228-6aa5-49d6-b26b-e1caa50f1212\") " pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"
Jul 24 18:13:13.668302 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.668277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxbc2\" (UniqueName: \"kubernetes.io/projected/822d7228-6aa5-49d6-b26b-e1caa50f1212-kube-api-access-nxbc2\") pod \"payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj\" (UID: \"822d7228-6aa5-49d6-b26b-e1caa50f1212\") " pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"
Jul 24 18:13:13.668625 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.668607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7dg\" (UniqueName: \"kubernetes.io/projected/89615b27-98e1-41c7-a530-2c2ab0abee03-kube-api-access-nw7dg\") pod \"payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf\" (UID: \"89615b27-98e1-41c7-a530-2c2ab0abee03\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"
Jul 24 18:13:13.678122 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.678094    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"]
Jul 24 18:13:13.681224 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:13:13.681201    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3316f5_b443_4df4_bdaa_bcd3e8f438b9.slice/crio-cf75e414ed6cc1c4f48e743aec9a4ca34e1ac2f799008e6abec5a8924694d400 WatchSource:0}: Error finding container cf75e414ed6cc1c4f48e743aec9a4ca34e1ac2f799008e6abec5a8924694d400: Status 404 returned error can't find the container with id cf75e414ed6cc1c4f48e743aec9a4ca34e1ac2f799008e6abec5a8924694d400
Jul 24 18:13:13.682695 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.682678    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:13:13.787984 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.787951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"
Jul 24 18:13:13.810924 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.810901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"
Jul 24 18:13:13.924594 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.924512    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"]
Jul 24 18:13:13.926809 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:13:13.926779    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89615b27_98e1_41c7_a530_2c2ab0abee03.slice/crio-623bd1784092a7fef30e69cb2282025f82b8d486097465caedef3e7b1e247931 WatchSource:0}: Error finding container 623bd1784092a7fef30e69cb2282025f82b8d486097465caedef3e7b1e247931: Status 404 returned error can't find the container with id 623bd1784092a7fef30e69cb2282025f82b8d486097465caedef3e7b1e247931
Jul 24 18:13:13.950426 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:13.950404    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"]
Jul 24 18:13:13.957621 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:13:13.957594    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822d7228_6aa5_49d6_b26b_e1caa50f1212.slice/crio-aaec7942622ee33a52325c18987d521053c903ad525ea70cb2db6b25b5798e4b WatchSource:0}: Error finding container aaec7942622ee33a52325c18987d521053c903ad525ea70cb2db6b25b5798e4b: Status 404 returned error can't find the container with id aaec7942622ee33a52325c18987d521053c903ad525ea70cb2db6b25b5798e4b
Jul 24 18:13:14.558755 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:14.558722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz" event={"ID":"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9","Type":"ContainerStarted","Data":"756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5"}
Jul 24 18:13:14.559185 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:14.558785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz" event={"ID":"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9","Type":"ContainerStarted","Data":"cf75e414ed6cc1c4f48e743aec9a4ca34e1ac2f799008e6abec5a8924694d400"}
Jul 24 18:13:14.559185 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:14.558815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:14.560421 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:14.560394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj" event={"ID":"822d7228-6aa5-49d6-b26b-e1caa50f1212","Type":"ContainerStarted","Data":"93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5"}
Jul 24 18:13:14.560537 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:14.560446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj" event={"ID":"822d7228-6aa5-49d6-b26b-e1caa50f1212","Type":"ContainerStarted","Data":"aaec7942622ee33a52325c18987d521053c903ad525ea70cb2db6b25b5798e4b"}
Jul 24 18:13:14.560537 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:14.560478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"
Jul 24 18:13:14.561957 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:14.561926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf" event={"ID":"89615b27-98e1-41c7-a530-2c2ab0abee03","Type":"ContainerStarted","Data":"e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950"}
Jul 24 18:13:14.562077 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:14.561972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf" event={"ID":"89615b27-98e1-41c7-a530-2c2ab0abee03","Type":"ContainerStarted","Data":"623bd1784092a7fef30e69cb2282025f82b8d486097465caedef3e7b1e247931"}
Jul 24 18:13:14.576822 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:14.576781    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz" podStartSLOduration=1.78997591 podStartE2EDuration="2.576767793s" totalImagesPullingTime="786.791883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:13:14.575295328 +0000 UTC m=+2208.091806364" watchObservedRunningTime="2026-07-24 18:13:14.576767793 +0000 UTC m=+2208.093278830"
Jul 24 18:13:14.592060 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:14.592019    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf" podStartSLOduration=1.5920056 podStartE2EDuration="1.5920056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:13:14.59065219 +0000 UTC m=+2208.107163231" watchObservedRunningTime="2026-07-24 18:13:14.5920056 +0000 UTC m=+2208.108516636"
Jul 24 18:13:14.606121 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:14.606083    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj" podStartSLOduration=1.606070462 podStartE2EDuration="1.606070462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:13:14.604949419 +0000 UTC m=+2208.121460457" watchObservedRunningTime="2026-07-24 18:13:14.606070462 +0000 UTC m=+2208.122581499"
Jul 24 18:13:15.565840 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:15.565801    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"
Jul 24 18:13:17.076081 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:17.076051    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:18.385900 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:18.385855    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:18.486294 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:18.486260    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:20.572710 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:20.572678    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:13:25.567750 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:25.567707    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"
Jul 24 18:13:25.609904 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:25.609871    2581 generic.go:356] "Generic (PLEG): container finished" podID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerID="e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f" exitCode=0
Jul 24 18:13:25.610055 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:25.609945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6" event={"ID":"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41","Type":"ContainerDied","Data":"e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f"}
Jul 24 18:13:25.610347 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:25.610333    2581 scope.go:117] "RemoveContainer" containerID="e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f"
Jul 24 18:13:25.611349 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:25.611325    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:13:25.611410 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:25.611383    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:13:25.611446 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:25.611424    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:13:26.571519 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:26.571485    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"
Jul 24 18:13:26.618781 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:26.618745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6" event={"ID":"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41","Type":"ContainerStarted","Data":"4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9"}
Jul 24 18:13:26.992074 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:26.991990    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:26.993049 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:26.993028    2581 patch_prober.go:28] interesting pod/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Jul 24 18:13:26.993104 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:26.993072    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6" podUID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Jul 24 18:13:27.547204 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:27.547159    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:27.601281 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:27.601248    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:13:27.993071 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:27.993039    2581 patch_prober.go:28] interesting pod/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Jul 24 18:13:27.993242 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:27.993095    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6" podUID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Jul 24 18:13:28.992428 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:28.992396    2581 patch_prober.go:28] interesting pod/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Jul 24 18:13:28.992884 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:28.992445    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6" podUID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Jul 24 18:13:29.992911 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:29.992880    2581 patch_prober.go:28] interesting pod/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Jul 24 18:13:29.993311 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:29.992931    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6" podUID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Jul 24 18:13:30.996381 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:30.996348    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:30.997444 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:30.997423    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:13:51.692097 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:51.692020    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m"]
Jul 24 18:13:51.696478 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:51.696460    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m"
Jul 24 18:13:51.698909 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:51.698890    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v7cm8\""
Jul 24 18:13:51.702247 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:51.702221    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m"]
Jul 24 18:13:51.889960 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:51.889923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26072f58-fa5c-48f1-94fb-5d0900d5a82d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m\" (UID: \"26072f58-fa5c-48f1-94fb-5d0900d5a82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m"
Jul 24 18:13:51.990625 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:51.990514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26072f58-fa5c-48f1-94fb-5d0900d5a82d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m\" (UID: \"26072f58-fa5c-48f1-94fb-5d0900d5a82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m"
Jul 24 18:13:51.991119 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:51.991098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26072f58-fa5c-48f1-94fb-5d0900d5a82d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m\" (UID: \"26072f58-fa5c-48f1-94fb-5d0900d5a82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m"
Jul 24 18:13:52.008656 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:52.008635    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m"
Jul 24 18:13:52.137082 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:52.137054    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m"]
Jul 24 18:13:52.139037 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:13:52.139014    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26072f58_fa5c_48f1_94fb_5d0900d5a82d.slice/crio-d413c4c891a5e18ed47b363a39af24dbd5d2e614559bb4777e5533f94ad2b459 WatchSource:0}: Error finding container d413c4c891a5e18ed47b363a39af24dbd5d2e614559bb4777e5533f94ad2b459: Status 404 returned error can't find the container with id d413c4c891a5e18ed47b363a39af24dbd5d2e614559bb4777e5533f94ad2b459
Jul 24 18:13:52.732319 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:52.732285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m" event={"ID":"26072f58-fa5c-48f1-94fb-5d0900d5a82d","Type":"ContainerStarted","Data":"fd7d34cc529d77b4f82d48e176b5e5417dd80881c7369a44a76df660af2e12ed"}
Jul 24 18:13:52.732712 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:52.732335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m" event={"ID":"26072f58-fa5c-48f1-94fb-5d0900d5a82d","Type":"ContainerStarted","Data":"d413c4c891a5e18ed47b363a39af24dbd5d2e614559bb4777e5533f94ad2b459"}
Jul 24 18:13:52.747735 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:52.747687    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m" podStartSLOduration=1.7476711090000001 podStartE2EDuration="1.747671109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:13:52.746501554 +0000 UTC m=+2246.263012589" watchObservedRunningTime="2026-07-24 18:13:52.747671109 +0000 UTC m=+2246.264182149"
Jul 24 18:13:53.737733 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:53.737699    2581 generic.go:356] "Generic (PLEG): container finished" podID="26072f58-fa5c-48f1-94fb-5d0900d5a82d" containerID="fd7d34cc529d77b4f82d48e176b5e5417dd80881c7369a44a76df660af2e12ed" exitCode=0
Jul 24 18:13:53.737733 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:53.737738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m" event={"ID":"26072f58-fa5c-48f1-94fb-5d0900d5a82d","Type":"ContainerDied","Data":"fd7d34cc529d77b4f82d48e176b5e5417dd80881c7369a44a76df660af2e12ed"}
Jul 24 18:13:54.872722 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:54.872694    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m"
Jul 24 18:13:54.916315 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:54.916281    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26072f58-fa5c-48f1-94fb-5d0900d5a82d-maas-api-service-ca\") pod \"26072f58-fa5c-48f1-94fb-5d0900d5a82d\" (UID: \"26072f58-fa5c-48f1-94fb-5d0900d5a82d\") "
Jul 24 18:13:54.916611 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:54.916590    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26072f58-fa5c-48f1-94fb-5d0900d5a82d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26072f58-fa5c-48f1-94fb-5d0900d5a82d" (UID: "26072f58-fa5c-48f1-94fb-5d0900d5a82d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:13:55.016966 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:55.016905    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26072f58-fa5c-48f1-94fb-5d0900d5a82d-maas-api-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:13:55.747143 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:55.747105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m" event={"ID":"26072f58-fa5c-48f1-94fb-5d0900d5a82d","Type":"ContainerDied","Data":"d413c4c891a5e18ed47b363a39af24dbd5d2e614559bb4777e5533f94ad2b459"}
Jul 24 18:13:55.747143 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:55.747141    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m"
Jul 24 18:13:55.747355 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:13:55.747165    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d413c4c891a5e18ed47b363a39af24dbd5d2e614559bb4777e5533f94ad2b459"
Jul 24 18:14:07.294690 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.294650    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"]
Jul 24 18:14:07.295092 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.294965    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz" podUID="8d3316f5-b443-4df4-bdaa-bcd3e8f438b9" containerName="maas-api" containerID="cri-o://756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5" gracePeriod=30
Jul 24 18:14:07.335096 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.335071    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:07.359211 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.359152    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"]
Jul 24 18:14:07.359419 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.359395    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj" podUID="822d7228-6aa5-49d6-b26b-e1caa50f1212" containerName="payload-processing" containerID="cri-o://93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5" gracePeriod=30
Jul 24 18:14:07.362649 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.362609    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"]
Jul 24 18:14:07.362876 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.362829    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf" podUID="89615b27-98e1-41c7-a530-2c2ab0abee03" containerName="payload-pre-processing" containerID="cri-o://e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950" gracePeriod=30
Jul 24 18:14:07.435052 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.435016    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:07.496694 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.496663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:07.634440 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.634411    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"
Jul 24 18:14:07.661346 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.661321    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"
Jul 24 18:14:07.670139 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.670119    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:14:07.707256 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.707227    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-openshift-service-ca\") pod \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") "
Jul 24 18:14:07.707402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.707275    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw7dg\" (UniqueName: \"kubernetes.io/projected/89615b27-98e1-41c7-a530-2c2ab0abee03-kube-api-access-nw7dg\") pod \"89615b27-98e1-41c7-a530-2c2ab0abee03\" (UID: \"89615b27-98e1-41c7-a530-2c2ab0abee03\") "
Jul 24 18:14:07.707402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.707304    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89615b27-98e1-41c7-a530-2c2ab0abee03-plugins-config-volume\") pod \"89615b27-98e1-41c7-a530-2c2ab0abee03\" (UID: \"89615b27-98e1-41c7-a530-2c2ab0abee03\") "
Jul 24 18:14:07.707402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.707396    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822d7228-6aa5-49d6-b26b-e1caa50f1212-plugins-config-volume\") pod \"822d7228-6aa5-49d6-b26b-e1caa50f1212\" (UID: \"822d7228-6aa5-49d6-b26b-e1caa50f1212\") "
Jul 24 18:14:07.707535 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.707452    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-maas-api-tls\") pod \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") "
Jul 24 18:14:07.707535 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.707487    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5n57\" (UniqueName: \"kubernetes.io/projected/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-kube-api-access-r5n57\") pod \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\" (UID: \"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9\") "
Jul 24 18:14:07.707535 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.707510    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxbc2\" (UniqueName: \"kubernetes.io/projected/822d7228-6aa5-49d6-b26b-e1caa50f1212-kube-api-access-nxbc2\") pod \"822d7228-6aa5-49d6-b26b-e1caa50f1212\" (UID: \"822d7228-6aa5-49d6-b26b-e1caa50f1212\") "
Jul 24 18:14:07.707730 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.707703    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d3316f5-b443-4df4-bdaa-bcd3e8f438b9" (UID: "8d3316f5-b443-4df4-bdaa-bcd3e8f438b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:14:07.707814 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.707754    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89615b27-98e1-41c7-a530-2c2ab0abee03-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89615b27-98e1-41c7-a530-2c2ab0abee03" (UID: "89615b27-98e1-41c7-a530-2c2ab0abee03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:14:07.707814 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.707771    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:14:07.707814 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.707792    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822d7228-6aa5-49d6-b26b-e1caa50f1212-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "822d7228-6aa5-49d6-b26b-e1caa50f1212" (UID: "822d7228-6aa5-49d6-b26b-e1caa50f1212"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:14:07.709624 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.709597    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822d7228-6aa5-49d6-b26b-e1caa50f1212-kube-api-access-nxbc2" (OuterVolumeSpecName: "kube-api-access-nxbc2") pod "822d7228-6aa5-49d6-b26b-e1caa50f1212" (UID: "822d7228-6aa5-49d6-b26b-e1caa50f1212"). InnerVolumeSpecName "kube-api-access-nxbc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:14:07.709755 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.709638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89615b27-98e1-41c7-a530-2c2ab0abee03-kube-api-access-nw7dg" (OuterVolumeSpecName: "kube-api-access-nw7dg") pod "89615b27-98e1-41c7-a530-2c2ab0abee03" (UID: "89615b27-98e1-41c7-a530-2c2ab0abee03"). InnerVolumeSpecName "kube-api-access-nw7dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:14:07.710107 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.710085    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d3316f5-b443-4df4-bdaa-bcd3e8f438b9" (UID: "8d3316f5-b443-4df4-bdaa-bcd3e8f438b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:14:07.710204 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.710099    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-kube-api-access-r5n57" (OuterVolumeSpecName: "kube-api-access-r5n57") pod "8d3316f5-b443-4df4-bdaa-bcd3e8f438b9" (UID: "8d3316f5-b443-4df4-bdaa-bcd3e8f438b9"). InnerVolumeSpecName "kube-api-access-r5n57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:14:07.798846 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.798809    2581 generic.go:356] "Generic (PLEG): container finished" podID="89615b27-98e1-41c7-a530-2c2ab0abee03" containerID="e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950" exitCode=0
Jul 24 18:14:07.798978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.798859    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"
Jul 24 18:14:07.798978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.798891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf" event={"ID":"89615b27-98e1-41c7-a530-2c2ab0abee03","Type":"ContainerDied","Data":"e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950"}
Jul 24 18:14:07.798978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.798943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf" event={"ID":"89615b27-98e1-41c7-a530-2c2ab0abee03","Type":"ContainerDied","Data":"623bd1784092a7fef30e69cb2282025f82b8d486097465caedef3e7b1e247931"}
Jul 24 18:14:07.798978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.798973    2581 scope.go:117] "RemoveContainer" containerID="e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950"
Jul 24 18:14:07.800116 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.800098    2581 generic.go:356] "Generic (PLEG): container finished" podID="8d3316f5-b443-4df4-bdaa-bcd3e8f438b9" containerID="756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5" exitCode=0
Jul 24 18:14:07.800212 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.800146    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"
Jul 24 18:14:07.800212 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.800179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz" event={"ID":"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9","Type":"ContainerDied","Data":"756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5"}
Jul 24 18:14:07.800348 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.800221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz" event={"ID":"8d3316f5-b443-4df4-bdaa-bcd3e8f438b9","Type":"ContainerDied","Data":"cf75e414ed6cc1c4f48e743aec9a4ca34e1ac2f799008e6abec5a8924694d400"}
Jul 24 18:14:07.801586 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.801541    2581 generic.go:356] "Generic (PLEG): container finished" podID="822d7228-6aa5-49d6-b26b-e1caa50f1212" containerID="93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5" exitCode=0
Jul 24 18:14:07.801686 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.801602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj" event={"ID":"822d7228-6aa5-49d6-b26b-e1caa50f1212","Type":"ContainerDied","Data":"93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5"}
Jul 24 18:14:07.801686 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.801624    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"
Jul 24 18:14:07.801686 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.801643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj" event={"ID":"822d7228-6aa5-49d6-b26b-e1caa50f1212","Type":"ContainerDied","Data":"aaec7942622ee33a52325c18987d521053c903ad525ea70cb2db6b25b5798e4b"}
Jul 24 18:14:07.807938 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.807921    2581 scope.go:117] "RemoveContainer" containerID="e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950"
Jul 24 18:14:07.808209 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:14:07.808183    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950\": container with ID starting with e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950 not found: ID does not exist" containerID="e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950"
Jul 24 18:14:07.808385 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.808214    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950"} err="failed to get container status \"e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950\": rpc error: code = NotFound desc = could not find container \"e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950\": container with ID starting with e005ecbdcd94b5e4e1d4e30ee7790672e1ffe4e1fdc96ace89d70499785bf950 not found: ID does not exist"
Jul 24 18:14:07.808385 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.808231    2581 scope.go:117] "RemoveContainer" containerID="756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5"
Jul 24 18:14:07.808385 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.808230    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7dg\" (UniqueName: \"kubernetes.io/projected/89615b27-98e1-41c7-a530-2c2ab0abee03-kube-api-access-nw7dg\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:14:07.808385 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.808252    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89615b27-98e1-41c7-a530-2c2ab0abee03-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:14:07.808385 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.808266    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822d7228-6aa5-49d6-b26b-e1caa50f1212-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:14:07.808385 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.808280    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:14:07.808385 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.808293    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5n57\" (UniqueName: \"kubernetes.io/projected/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9-kube-api-access-r5n57\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:14:07.808385 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.808306    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxbc2\" (UniqueName: \"kubernetes.io/projected/822d7228-6aa5-49d6-b26b-e1caa50f1212-kube-api-access-nxbc2\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:14:07.816594 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.816556    2581 scope.go:117] "RemoveContainer" containerID="756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5"
Jul 24 18:14:07.816818 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:14:07.816798    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5\": container with ID starting with 756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5 not found: ID does not exist" containerID="756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5"
Jul 24 18:14:07.816885 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.816823    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5"} err="failed to get container status \"756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5\": rpc error: code = NotFound desc = could not find container \"756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5\": container with ID starting with 756cd500e8887bd7a931e67f382e51679a3302ea10cffe940daa4a9551f629d5 not found: ID does not exist"
Jul 24 18:14:07.816885 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.816839    2581 scope.go:117] "RemoveContainer" containerID="93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5"
Jul 24 18:14:07.823070 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.823021    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"]
Jul 24 18:14:07.825072 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.825050    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a046f5ba-5566bddb88-g8stf"]
Jul 24 18:14:07.825293 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.825279    2581 scope.go:117] "RemoveContainer" containerID="93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5"
Jul 24 18:14:07.825549 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:14:07.825533    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5\": container with ID starting with 93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5 not found: ID does not exist" containerID="93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5"
Jul 24 18:14:07.825628 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.825553    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5"} err="failed to get container status \"93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5\": rpc error: code = NotFound desc = could not find container \"93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5\": container with ID starting with 93bd907edf3c10b0e2e5744c637ca4b7a2ad29ac869523b32a482c1da4bf44c5 not found: ID does not exist"
Jul 24 18:14:07.834331 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.834311    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"]
Jul 24 18:14:07.838898 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.838875    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a046f5ba-5c98589b5d-fhrqz"]
Jul 24 18:14:07.847971 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.847951    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"]
Jul 24 18:14:07.851413 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:07.851393    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a046f5ba-76f9bb8c74-97ktj"]
Jul 24 18:14:08.957624 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:08.957562    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="822d7228-6aa5-49d6-b26b-e1caa50f1212" path="/var/lib/kubelet/pods/822d7228-6aa5-49d6-b26b-e1caa50f1212/volumes"
Jul 24 18:14:08.958115 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:08.958101    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89615b27-98e1-41c7-a530-2c2ab0abee03" path="/var/lib/kubelet/pods/89615b27-98e1-41c7-a530-2c2ab0abee03/volumes"
Jul 24 18:14:08.958499 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:08.958478    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d3316f5-b443-4df4-bdaa-bcd3e8f438b9" path="/var/lib/kubelet/pods/8d3316f5-b443-4df4-bdaa-bcd3e8f438b9/volumes"
Jul 24 18:14:11.792163 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:11.792130    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m"]
Jul 24 18:14:11.794449 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:11.794422    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a046f5ba-mv52m"]
Jul 24 18:14:11.801418 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:11.801394    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:12.956814 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:12.956781    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26072f58-fa5c-48f1-94fb-5d0900d5a82d" path="/var/lib/kubelet/pods/26072f58-fa5c-48f1-94fb-5d0900d5a82d/volumes"
Jul 24 18:14:15.145628 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:15.145594    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:15.244336 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:15.244303    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:20.311413 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:20.311381    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:56.758509 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:56.758460    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:14:57.895420 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:14:57.895384    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:00.144679 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.144647    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748615-ldj8z"]
Jul 24 18:15:00.145135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.144916    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26072f58-fa5c-48f1-94fb-5d0900d5a82d" containerName="revoke-keys"
Jul 24 18:15:00.145135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.144927    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="26072f58-fa5c-48f1-94fb-5d0900d5a82d" containerName="revoke-keys"
Jul 24 18:15:00.145135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.144943    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d3316f5-b443-4df4-bdaa-bcd3e8f438b9" containerName="maas-api"
Jul 24 18:15:00.145135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.144948    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d3316f5-b443-4df4-bdaa-bcd3e8f438b9" containerName="maas-api"
Jul 24 18:15:00.145135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.144957    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822d7228-6aa5-49d6-b26b-e1caa50f1212" containerName="payload-processing"
Jul 24 18:15:00.145135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.144962    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="822d7228-6aa5-49d6-b26b-e1caa50f1212" containerName="payload-processing"
Jul 24 18:15:00.145135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.144974    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89615b27-98e1-41c7-a530-2c2ab0abee03" containerName="payload-pre-processing"
Jul 24 18:15:00.145135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.144980    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89615b27-98e1-41c7-a530-2c2ab0abee03" containerName="payload-pre-processing"
Jul 24 18:15:00.145135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.145024    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d3316f5-b443-4df4-bdaa-bcd3e8f438b9" containerName="maas-api"
Jul 24 18:15:00.145135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.145032    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="26072f58-fa5c-48f1-94fb-5d0900d5a82d" containerName="revoke-keys"
Jul 24 18:15:00.145135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.145038    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="822d7228-6aa5-49d6-b26b-e1caa50f1212" containerName="payload-processing"
Jul 24 18:15:00.145135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.145044    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89615b27-98e1-41c7-a530-2c2ab0abee03" containerName="payload-pre-processing"
Jul 24 18:15:00.147920 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.147903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-ldj8z"
Jul 24 18:15:00.150784 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.150765    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v7cm8\""
Jul 24 18:15:00.156839 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.156822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-ldj8z"
Jul 24 18:15:00.161223 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.161201    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748615-ldj8z"]
Jul 24 18:15:00.304616 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:00.304513    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748615-ldj8z"]
Jul 24 18:15:00.307164 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:15:00.307137    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc099e9d6_dc3a_4750_8d02_2e106b7c48bc.slice/crio-00543d25affab34cf8a1792ff70a7758cbd9c7c60fff3aadb6401814b1015c82 WatchSource:0}: Error finding container 00543d25affab34cf8a1792ff70a7758cbd9c7c60fff3aadb6401814b1015c82: Status 404 returned error can't find the container with id 00543d25affab34cf8a1792ff70a7758cbd9c7c60fff3aadb6401814b1015c82
Jul 24 18:15:01.032026 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:01.031992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-ldj8z" event={"ID":"c099e9d6-dc3a-4750-8d02-2e106b7c48bc","Type":"ContainerStarted","Data":"e4f6bdf0f20790b2111ec00061d70c6af917053aea4a5478f854d8b25a2a7119"}
Jul 24 18:15:01.032203 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:01.032045    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-ldj8z" event={"ID":"c099e9d6-dc3a-4750-8d02-2e106b7c48bc","Type":"ContainerStarted","Data":"00543d25affab34cf8a1792ff70a7758cbd9c7c60fff3aadb6401814b1015c82"}
Jul 24 18:15:01.048561 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:01.048510    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-ldj8z" podStartSLOduration=1.048491592 podStartE2EDuration="1.048491592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:15:01.047264634 +0000 UTC m=+2314.563775671" watchObservedRunningTime="2026-07-24 18:15:01.048491592 +0000 UTC m=+2314.565002632"
Jul 24 18:15:02.037191 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:02.037159    2581 generic.go:356] "Generic (PLEG): container finished" podID="c099e9d6-dc3a-4750-8d02-2e106b7c48bc" containerID="e4f6bdf0f20790b2111ec00061d70c6af917053aea4a5478f854d8b25a2a7119" exitCode=0
Jul 24 18:15:02.037601 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:02.037246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-ldj8z" event={"ID":"c099e9d6-dc3a-4750-8d02-2e106b7c48bc","Type":"ContainerDied","Data":"e4f6bdf0f20790b2111ec00061d70c6af917053aea4a5478f854d8b25a2a7119"}
Jul 24 18:15:03.092764 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:03.092723    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:03.169378 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:03.169352    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-ldj8z"
Jul 24 18:15:03.190019 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:03.189989    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:03.357700 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:03.357666    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"]
Jul 24 18:15:03.357923 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:03.357899    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6" podUID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerName="istio-proxy" containerID="cri-o://4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9" gracePeriod=30
Jul 24 18:15:03.389539 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:03.389509    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:03.626203 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:03.626127    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:04.049635 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:04.049555    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-ldj8z"
Jul 24 18:15:04.049763 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:04.049555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-ldj8z" event={"ID":"c099e9d6-dc3a-4750-8d02-2e106b7c48bc","Type":"ContainerDied","Data":"00543d25affab34cf8a1792ff70a7758cbd9c7c60fff3aadb6401814b1015c82"}
Jul 24 18:15:04.049763 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:04.049703    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00543d25affab34cf8a1792ff70a7758cbd9c7c60fff3aadb6401814b1015c82"
Jul 24 18:15:04.779497 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:04.779463    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:04.922840 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:04.922807    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:05.030742 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:05.030662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:05.739018 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:05.738986    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:05.839152 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:05.839120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:05.939229 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:05.939199    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:07.305602 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:07.305556    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:08.605926 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.605903    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:15:08.703290 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703209    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzgx2\" (UniqueName: \"kubernetes.io/projected/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-kube-api-access-lzgx2\") pod \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") "
Jul 24 18:15:08.703290 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-data\") pod \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") "
Jul 24 18:15:08.703290 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703273    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-workload-socket\") pod \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") "
Jul 24 18:15:08.703560 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703297    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-credential-socket\") pod \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") "
Jul 24 18:15:08.703560 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703314    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-workload-certs\") pod \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") "
Jul 24 18:15:08.703560 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703340    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istiod-ca-cert\") pod \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") "
Jul 24 18:15:08.703560 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703375    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-token\") pod \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") "
Jul 24 18:15:08.703560 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703399    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-podinfo\") pod \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") "
Jul 24 18:15:08.703560 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703422    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-envoy\") pod \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\" (UID: \"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41\") "
Jul 24 18:15:08.703867 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703558    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" (UID: "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:15:08.703867 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703657    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" (UID: "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:15:08.703977 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703889    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" (UID: "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:15:08.703977 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.703740    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" (UID: "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:15:08.704351 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.704324    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-data" (OuterVolumeSpecName: "istio-data") pod "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" (UID: "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:15:08.705590 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.705555    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-kube-api-access-lzgx2" (OuterVolumeSpecName: "kube-api-access-lzgx2") pod "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" (UID: "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41"). InnerVolumeSpecName "kube-api-access-lzgx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:15:08.705692 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.705642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" (UID: "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 18:15:08.705692 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.705676    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" (UID: "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:15:08.705772 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.705716    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-token" (OuterVolumeSpecName: "istio-token") pod "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" (UID: "1de7e79f-015a-4f9e-bdbd-b19f5f3afd41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:15:08.803915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.803889    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:15:08.803915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.803911    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:15:08.803915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.803920    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:15:08.804108 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.803928    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:15:08.804108 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.803938    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:15:08.804108 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.803945    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:15:08.804108 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.803952    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:15:08.804108 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.803959    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:15:08.804108 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:08.803967    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzgx2\" (UniqueName: \"kubernetes.io/projected/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41-kube-api-access-lzgx2\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:15:09.072037 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:09.072005    2581 generic.go:356] "Generic (PLEG): container finished" podID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerID="4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9" exitCode=0
Jul 24 18:15:09.072201 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:09.072080    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"
Jul 24 18:15:09.072201 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:09.072094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6" event={"ID":"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41","Type":"ContainerDied","Data":"4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9"}
Jul 24 18:15:09.072201 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:09.072165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6" event={"ID":"1de7e79f-015a-4f9e-bdbd-b19f5f3afd41","Type":"ContainerDied","Data":"b8f84e6ae26848941e561bd3bb66700aba1778cc58804d6e16e2d34273715562"}
Jul 24 18:15:09.072201 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:09.072199    2581 scope.go:117] "RemoveContainer" containerID="4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9"
Jul 24 18:15:09.081063 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:09.081043    2581 scope.go:117] "RemoveContainer" containerID="e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f"
Jul 24 18:15:09.088607 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:09.088580    2581 scope.go:117] "RemoveContainer" containerID="4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9"
Jul 24 18:15:09.088919 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:15:09.088897    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9\": container with ID starting with 4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9 not found: ID does not exist" containerID="4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9"
Jul 24 18:15:09.089010 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:09.088932    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9"} err="failed to get container status \"4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9\": rpc error: code = NotFound desc = could not find container \"4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9\": container with ID starting with 4b5d61c41b44a240f3c8c9b274ed2452b10ebb027d207147429379853eed5bc9 not found: ID does not exist"
Jul 24 18:15:09.089010 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:09.088956    2581 scope.go:117] "RemoveContainer" containerID="e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f"
Jul 24 18:15:09.089125 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:09.089108    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"]
Jul 24 18:15:09.089207 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:15:09.089191    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f\": container with ID starting with e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f not found: ID does not exist" containerID="e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f"
Jul 24 18:15:09.089249 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:09.089215    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f"} err="failed to get container status \"e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f\": rpc error: code = NotFound desc = could not find container \"e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f\": container with ID starting with e5548d676f1c329ff672c837772c04ebcd6fb082f993a7b755523b9d7fdde34f not found: ID does not exist"
Jul 24 18:15:09.098115 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:09.098095    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a046f5ba-openshift-default-7495bd7b9b-znrp6"]
Jul 24 18:15:10.957543 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:10.957503    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" path="/var/lib/kubelet/pods/1de7e79f-015a-4f9e-bdbd-b19f5f3afd41/volumes"
Jul 24 18:15:12.961916 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:12.961875    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:13.026075 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:13.026040    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:13.645241 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:13.645209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:19.591464 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:19.591375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:19.695724 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:19.695686    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:19.827211 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:19.827170    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:27.074454 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:27.074424    2581 scope.go:117] "RemoveContainer" containerID="136c1b6ce76d26b0753aa9b58345719b30e874c07f5fc900f3bee173dd383e3b"
Jul 24 18:15:30.390606 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:30.390547    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:30.755987 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:30.755894    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:40.860983 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:40.860952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:51.781968 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:51.781928    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:57.990480 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:57.990436    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:58.089822 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.089790    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:58.372204 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.372157    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:58.723257 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.723169    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"]
Jul 24 18:15:58.723554 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.723536    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerName="istio-proxy"
Jul 24 18:15:58.723663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.723557    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerName="istio-proxy"
Jul 24 18:15:58.723663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.723589    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerName="istio-proxy"
Jul 24 18:15:58.723663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.723598    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerName="istio-proxy"
Jul 24 18:15:58.723663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.723616    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c099e9d6-dc3a-4750-8d02-2e106b7c48bc" containerName="cleanup"
Jul 24 18:15:58.723663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.723625    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c099e9d6-dc3a-4750-8d02-2e106b7c48bc" containerName="cleanup"
Jul 24 18:15:58.723916 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.723704    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c099e9d6-dc3a-4750-8d02-2e106b7c48bc" containerName="cleanup"
Jul 24 18:15:58.723916 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.723717    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerName="istio-proxy"
Jul 24 18:15:58.723916 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.723727    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de7e79f-015a-4f9e-bdbd-b19f5f3afd41" containerName="istio-proxy"
Jul 24 18:15:58.726827 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.726805    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.729560 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.729534    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b3ffbc-openshift-default-dockercfg-8wld8\""
Jul 24 18:15:58.739822 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.739798    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"]
Jul 24 18:15:58.753878 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.753849    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:58.793365 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.793332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-envoy\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.793484 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.793381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-token\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.793484 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.793431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-credential-socket\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.793484 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.793455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-podinfo\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.793484 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.793474    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-workload-certs\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.793704 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.793542    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istiod-ca-cert\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.793704 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.793629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgr8b\" (UniqueName: \"kubernetes.io/projected/6152dbd8-b79e-4f90-88d4-a298a5772a3f-kube-api-access-cgr8b\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.793704 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.793650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-workload-socket\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.793704 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.793665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-data\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.894807 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.894774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-workload-certs\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.894978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.894816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istiod-ca-cert\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.894978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.894856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgr8b\" (UniqueName: \"kubernetes.io/projected/6152dbd8-b79e-4f90-88d4-a298a5772a3f-kube-api-access-cgr8b\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.894978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.894875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-workload-socket\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.894978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.894892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-data\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.894978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.894929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-envoy\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.894978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.894955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-token\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.895281 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.894984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-credential-socket\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.895281 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.895008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-podinfo\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.895281 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.895236    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-workload-certs\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.895429 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.895403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-data\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.895491 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.895430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-workload-socket\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.895491 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.895458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-credential-socket\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.895701 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.895679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istiod-ca-cert\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.897287 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.897270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-envoy\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.897510 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.897493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-podinfo\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.902970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.902948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-token\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:58.903148 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:58.903123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgr8b\" (UniqueName: \"kubernetes.io/projected/6152dbd8-b79e-4f90-88d4-a298a5772a3f-kube-api-access-cgr8b\") pod \"e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") " pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:59.013116 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:59.013033    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:15:59.036617 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:59.036590    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:15:59.170137 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:59.170108    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"]
Jul 24 18:15:59.172601 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:15:59.172558    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6152dbd8_b79e_4f90_88d4_a298a5772a3f.slice/crio-f7d694c8c578051374c930adf346a068472fbe4c7f9366858e913f2dcd7a716e WatchSource:0}: Error finding container f7d694c8c578051374c930adf346a068472fbe4c7f9366858e913f2dcd7a716e: Status 404 returned error can't find the container with id f7d694c8c578051374c930adf346a068472fbe4c7f9366858e913f2dcd7a716e
Jul 24 18:15:59.174755 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:59.174714    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:15:59.174862 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:59.174799    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:15:59.174862 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:59.174838    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:15:59.281105 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:59.281030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" event={"ID":"6152dbd8-b79e-4f90-88d4-a298a5772a3f","Type":"ContainerStarted","Data":"c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331"}
Jul 24 18:15:59.281105 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:59.281089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" event={"ID":"6152dbd8-b79e-4f90-88d4-a298a5772a3f","Type":"ContainerStarted","Data":"f7d694c8c578051374c930adf346a068472fbe4c7f9366858e913f2dcd7a716e"}
Jul 24 18:15:59.301827 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:15:59.301780    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" podStartSLOduration=1.301765317 podStartE2EDuration="1.301765317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:15:59.300240491 +0000 UTC m=+2372.816751527" watchObservedRunningTime="2026-07-24 18:15:59.301765317 +0000 UTC m=+2372.818276354"
Jul 24 18:16:00.036769 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:00.036732    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:16:00.041830 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:00.041800    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:16:00.290115 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:00.290034    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:16:00.291077 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:00.291055    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:16:01.482293 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.482259    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:01.582832 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.582801    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:01.718072 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.718039    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"]
Jul 24 18:16:01.721387 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.721365    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:01.724616 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.724550    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:16:01.725522 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.725501    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:16:01.725657 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.725539    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fvm85\""
Jul 24 18:16:01.725657 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.725613    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b3ffbc-serving-cert\""
Jul 24 18:16:01.737344 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.737284    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"]
Jul 24 18:16:01.780101 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.780051    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:01.819028 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.819001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765a4862-2f9c-453f-b19e-387804a60459-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:01.819179 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.819040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765a4862-2f9c-453f-b19e-387804a60459-maas-api-tls\") pod \"maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:01.819179 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.819083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlwhp\" (UniqueName: \"kubernetes.io/projected/765a4862-2f9c-453f-b19e-387804a60459-kube-api-access-jlwhp\") pod \"maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:01.919917 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.919887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765a4862-2f9c-453f-b19e-387804a60459-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:01.920053 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.919925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765a4862-2f9c-453f-b19e-387804a60459-maas-api-tls\") pod \"maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:01.920053 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.919946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlwhp\" (UniqueName: \"kubernetes.io/projected/765a4862-2f9c-453f-b19e-387804a60459-kube-api-access-jlwhp\") pod \"maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:01.920128 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:16:01.920059    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-serving-cert: secret "maas-api-e2e-inf-a-b3ffbc-serving-cert" not found
Jul 24 18:16:01.920167 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:16:01.920144    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/765a4862-2f9c-453f-b19e-387804a60459-maas-api-tls podName:765a4862-2f9c-453f-b19e-387804a60459 nodeName:}" failed. No retries permitted until 2026-07-24 18:16:02.420120256 +0000 UTC m=+2375.936631277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/765a4862-2f9c-453f-b19e-387804a60459-maas-api-tls") pod "maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5" (UID: "765a4862-2f9c-453f-b19e-387804a60459") : secret "maas-api-e2e-inf-a-b3ffbc-serving-cert" not found
Jul 24 18:16:01.920607 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.920563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765a4862-2f9c-453f-b19e-387804a60459-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:01.929632 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:01.929615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlwhp\" (UniqueName: \"kubernetes.io/projected/765a4862-2f9c-453f-b19e-387804a60459-kube-api-access-jlwhp\") pod \"maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:02.186450 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.186421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:02.371808 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.371775    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:02.424111 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.424074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765a4862-2f9c-453f-b19e-387804a60459-maas-api-tls\") pod \"maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:02.426596 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.426555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765a4862-2f9c-453f-b19e-387804a60459-maas-api-tls\") pod \"maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:02.460205 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.460134    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:02.499296 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.499266    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"]
Jul 24 18:16:02.502424 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.502409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"
Jul 24 18:16:02.505809 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.505789    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b3ffbc\""
Jul 24 18:16:02.521958 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.521937    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"]
Jul 24 18:16:02.524679 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.524658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktn6\" (UniqueName: \"kubernetes.io/projected/8247fad9-1f8f-4005-9208-20b97fcad722-kube-api-access-pktn6\") pod \"payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs\" (UID: \"8247fad9-1f8f-4005-9208-20b97fcad722\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"
Jul 24 18:16:02.524768 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.524724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8247fad9-1f8f-4005-9208-20b97fcad722-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs\" (UID: \"8247fad9-1f8f-4005-9208-20b97fcad722\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"
Jul 24 18:16:02.560745 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.560720    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"]
Jul 24 18:16:02.563917 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.563901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"
Jul 24 18:16:02.567445 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.567430    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b3ffbc-dockercfg-65bwk\""
Jul 24 18:16:02.578212 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.578191    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"]
Jul 24 18:16:02.625849 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.625817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk9sv\" (UniqueName: \"kubernetes.io/projected/ac324830-205b-4fb6-8864-64c178853ccb-kube-api-access-kk9sv\") pod \"payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp\" (UID: \"ac324830-205b-4fb6-8864-64c178853ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"
Jul 24 18:16:02.626041 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.625870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac324830-205b-4fb6-8864-64c178853ccb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp\" (UID: \"ac324830-205b-4fb6-8864-64c178853ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"
Jul 24 18:16:02.626041 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.625982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8247fad9-1f8f-4005-9208-20b97fcad722-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs\" (UID: \"8247fad9-1f8f-4005-9208-20b97fcad722\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"
Jul 24 18:16:02.626140 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.626098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktn6\" (UniqueName: \"kubernetes.io/projected/8247fad9-1f8f-4005-9208-20b97fcad722-kube-api-access-pktn6\") pod \"payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs\" (UID: \"8247fad9-1f8f-4005-9208-20b97fcad722\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"
Jul 24 18:16:02.626724 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.626707    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8247fad9-1f8f-4005-9208-20b97fcad722-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs\" (UID: \"8247fad9-1f8f-4005-9208-20b97fcad722\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"
Jul 24 18:16:02.631745 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.631714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:02.635716 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.635457    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktn6\" (UniqueName: \"kubernetes.io/projected/8247fad9-1f8f-4005-9208-20b97fcad722-kube-api-access-pktn6\") pod \"payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs\" (UID: \"8247fad9-1f8f-4005-9208-20b97fcad722\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"
Jul 24 18:16:02.727291 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.727259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk9sv\" (UniqueName: \"kubernetes.io/projected/ac324830-205b-4fb6-8864-64c178853ccb-kube-api-access-kk9sv\") pod \"payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp\" (UID: \"ac324830-205b-4fb6-8864-64c178853ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"
Jul 24 18:16:02.727424 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.727300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac324830-205b-4fb6-8864-64c178853ccb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp\" (UID: \"ac324830-205b-4fb6-8864-64c178853ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"
Jul 24 18:16:02.727948 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.727930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac324830-205b-4fb6-8864-64c178853ccb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp\" (UID: \"ac324830-205b-4fb6-8864-64c178853ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"
Jul 24 18:16:02.736055 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.736030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk9sv\" (UniqueName: \"kubernetes.io/projected/ac324830-205b-4fb6-8864-64c178853ccb-kube-api-access-kk9sv\") pod \"payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp\" (UID: \"ac324830-205b-4fb6-8864-64c178853ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"
Jul 24 18:16:02.763720 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.763699    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"]
Jul 24 18:16:02.766033 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:16:02.766010    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765a4862_2f9c_453f_b19e_387804a60459.slice/crio-61d28e21fc3a8e436c2f5d4282204a49d492ac16a8c85fbe93a5c9be9b8221f1 WatchSource:0}: Error finding container 61d28e21fc3a8e436c2f5d4282204a49d492ac16a8c85fbe93a5c9be9b8221f1: Status 404 returned error can't find the container with id 61d28e21fc3a8e436c2f5d4282204a49d492ac16a8c85fbe93a5c9be9b8221f1
Jul 24 18:16:02.812885 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.812860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"
Jul 24 18:16:02.873212 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.873180    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"
Jul 24 18:16:02.950983 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:02.950956    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"]
Jul 24 18:16:02.952782 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:16:02.952756    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8247fad9_1f8f_4005_9208_20b97fcad722.slice/crio-347c99c5b9e13e04885ebe6b1256b721118f2cbf5acad528ed32d9bf95f94258 WatchSource:0}: Error finding container 347c99c5b9e13e04885ebe6b1256b721118f2cbf5acad528ed32d9bf95f94258: Status 404 returned error can't find the container with id 347c99c5b9e13e04885ebe6b1256b721118f2cbf5acad528ed32d9bf95f94258
Jul 24 18:16:03.016058 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:03.015922    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"]
Jul 24 18:16:03.018726 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:16:03.018697    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac324830_205b_4fb6_8864_64c178853ccb.slice/crio-5eacdc5a2dc40953f2fd1151e30e9507bb7fa02b9a992a277c4deaedb8d5bd9c WatchSource:0}: Error finding container 5eacdc5a2dc40953f2fd1151e30e9507bb7fa02b9a992a277c4deaedb8d5bd9c: Status 404 returned error can't find the container with id 5eacdc5a2dc40953f2fd1151e30e9507bb7fa02b9a992a277c4deaedb8d5bd9c
Jul 24 18:16:03.305625 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:03.305595    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5" event={"ID":"765a4862-2f9c-453f-b19e-387804a60459","Type":"ContainerStarted","Data":"61d28e21fc3a8e436c2f5d4282204a49d492ac16a8c85fbe93a5c9be9b8221f1"}
Jul 24 18:16:03.306995 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:03.306971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp" event={"ID":"ac324830-205b-4fb6-8864-64c178853ccb","Type":"ContainerStarted","Data":"8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05"}
Jul 24 18:16:03.307116 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:03.307013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp" event={"ID":"ac324830-205b-4fb6-8864-64c178853ccb","Type":"ContainerStarted","Data":"5eacdc5a2dc40953f2fd1151e30e9507bb7fa02b9a992a277c4deaedb8d5bd9c"}
Jul 24 18:16:03.307116 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:03.307080    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"
Jul 24 18:16:03.308340 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:03.308320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs" event={"ID":"8247fad9-1f8f-4005-9208-20b97fcad722","Type":"ContainerStarted","Data":"1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a"}
Jul 24 18:16:03.308423 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:03.308356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs" event={"ID":"8247fad9-1f8f-4005-9208-20b97fcad722","Type":"ContainerStarted","Data":"347c99c5b9e13e04885ebe6b1256b721118f2cbf5acad528ed32d9bf95f94258"}
Jul 24 18:16:03.308423 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:03.308414    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"
Jul 24 18:16:03.323040 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:03.323004    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp" podStartSLOduration=1.3229920609999999 podStartE2EDuration="1.322992061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:16:03.321457423 +0000 UTC m=+2376.837968460" watchObservedRunningTime="2026-07-24 18:16:03.322992061 +0000 UTC m=+2376.839503097"
Jul 24 18:16:03.339106 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:03.339056    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs" podStartSLOduration=1.339031731 podStartE2EDuration="1.339031731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:16:03.336787013 +0000 UTC m=+2376.853298119" watchObservedRunningTime="2026-07-24 18:16:03.339031731 +0000 UTC m=+2376.855542769"
Jul 24 18:16:04.314034 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:04.313992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5" event={"ID":"765a4862-2f9c-453f-b19e-387804a60459","Type":"ContainerStarted","Data":"89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375"}
Jul 24 18:16:04.314493 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:04.314359    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:04.331628 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:04.331561    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5" podStartSLOduration=2.826407939 podStartE2EDuration="3.331546047s" totalImagesPullingTime="505.138108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:16:04.329965923 +0000 UTC m=+2377.846476961" watchObservedRunningTime="2026-07-24 18:16:04.331546047 +0000 UTC m=+2377.848057083"
Jul 24 18:16:05.762394 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:05.762361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:06.183557 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.183519    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:06.509778 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.509703    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"]
Jul 24 18:16:06.513211 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.513192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.516352 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.516331    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-12361c-openshift-default-dockercfg-snmqt\""
Jul 24 18:16:06.540210 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.540185    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"]
Jul 24 18:16:06.550020 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.549992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:06.562450 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.562426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-workload-certs\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.562551 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.562521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72kv\" (UniqueName: \"kubernetes.io/projected/34ea2cb7-17a3-454d-b837-40c6484cc2fa-kube-api-access-w72kv\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.562663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.562646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-podinfo\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.562750 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.562728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-data\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.562812 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.562798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-workload-socket\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.562870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.562827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-envoy\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.562870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.562858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-credential-socket\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.562975 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.562893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-token\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.562975 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.562942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istiod-ca-cert\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.645168 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.645135    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:06.664124 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-podinfo\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.664124 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-data\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.664346 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-workload-socket\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.664346 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-envoy\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.664346 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-credential-socket\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.664346 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-token\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.664554 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istiod-ca-cert\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.664554 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-workload-certs\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.664554 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72kv\" (UniqueName: \"kubernetes.io/projected/34ea2cb7-17a3-454d-b837-40c6484cc2fa-kube-api-access-w72kv\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.664554 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-data\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.664554 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-workload-socket\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.664833 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664706    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-credential-socket\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.664900 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.664866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-workload-certs\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.665327 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.665299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istiod-ca-cert\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.666534 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.666506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-envoy\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.666849 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.666829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-podinfo\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.674285 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.674245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-token\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.679225 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.679192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72kv\" (UniqueName: \"kubernetes.io/projected/34ea2cb7-17a3-454d-b837-40c6484cc2fa-kube-api-access-w72kv\") pod \"e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") " pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.743193 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.743164    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:06.823814 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.823774    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:06.965253 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.965228    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"]
Jul 24 18:16:06.967528 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:16:06.967505    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ea2cb7_17a3_454d_b837_40c6484cc2fa.slice/crio-b44962b35354c5eb94083e3b8603a9181add66215434d120139ad5620b443a2b WatchSource:0}: Error finding container b44962b35354c5eb94083e3b8603a9181add66215434d120139ad5620b443a2b: Status 404 returned error can't find the container with id b44962b35354c5eb94083e3b8603a9181add66215434d120139ad5620b443a2b
Jul 24 18:16:06.969563 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.969537    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:16:06.969634 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.969608    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:16:06.969672 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:06.969637    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:16:07.330323 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:07.330283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" event={"ID":"34ea2cb7-17a3-454d-b837-40c6484cc2fa","Type":"ContainerStarted","Data":"ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa"}
Jul 24 18:16:07.330500 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:07.330345    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" event={"ID":"34ea2cb7-17a3-454d-b837-40c6484cc2fa","Type":"ContainerStarted","Data":"b44962b35354c5eb94083e3b8603a9181add66215434d120139ad5620b443a2b"}
Jul 24 18:16:07.366299 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:07.366242    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" podStartSLOduration=1.366224796 podStartE2EDuration="1.366224796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:16:07.357180222 +0000 UTC m=+2380.873691294" watchObservedRunningTime="2026-07-24 18:16:07.366224796 +0000 UTC m=+2380.882735831"
Jul 24 18:16:07.824818 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:07.824793    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:08.828611 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:08.828556    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:09.338955 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:09.338923    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:09.339850 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:09.339832    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:09.491467 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:09.491436    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:09.595269 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:09.595190    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:09.699097 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:09.699065    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:10.260764 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.260732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:10.435478 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.435451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:10.532787 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.532707    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:10.533663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.533644    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"]
Jul 24 18:16:10.536976 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.536961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"
Jul 24 18:16:10.539825 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.539807    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-12361c\""
Jul 24 18:16:10.546594 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.546555    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"]
Jul 24 18:16:10.568055 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.568029    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"]
Jul 24 18:16:10.571176 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.571161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"
Jul 24 18:16:10.573668 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.573648    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-12361c-dockercfg-snf7d\""
Jul 24 18:16:10.585778 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.585756    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"]
Jul 24 18:16:10.596448 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.596422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45j8p\" (UniqueName: \"kubernetes.io/projected/0d342795-84fc-4a18-9c69-efe23fb8dbcd-kube-api-access-45j8p\") pod \"payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22\" (UID: \"0d342795-84fc-4a18-9c69-efe23fb8dbcd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"
Jul 24 18:16:10.596549 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.596526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024fd37c-99a2-4edc-9f1a-fd857e002fbd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-12361c-5889644b5b-hq998\" (UID: \"024fd37c-99a2-4edc-9f1a-fd857e002fbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"
Jul 24 18:16:10.596627 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.596590    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvl5b\" (UniqueName: \"kubernetes.io/projected/024fd37c-99a2-4edc-9f1a-fd857e002fbd-kube-api-access-qvl5b\") pod \"payload-processing-e2e-inf-b-12361c-5889644b5b-hq998\" (UID: \"024fd37c-99a2-4edc-9f1a-fd857e002fbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"
Jul 24 18:16:10.596684 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.596664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d342795-84fc-4a18-9c69-efe23fb8dbcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22\" (UID: \"0d342795-84fc-4a18-9c69-efe23fb8dbcd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"
Jul 24 18:16:10.697813 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.697778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d342795-84fc-4a18-9c69-efe23fb8dbcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22\" (UID: \"0d342795-84fc-4a18-9c69-efe23fb8dbcd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"
Jul 24 18:16:10.697974 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.697830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45j8p\" (UniqueName: \"kubernetes.io/projected/0d342795-84fc-4a18-9c69-efe23fb8dbcd-kube-api-access-45j8p\") pod \"payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22\" (UID: \"0d342795-84fc-4a18-9c69-efe23fb8dbcd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"
Jul 24 18:16:10.697974 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.697898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024fd37c-99a2-4edc-9f1a-fd857e002fbd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-12361c-5889644b5b-hq998\" (UID: \"024fd37c-99a2-4edc-9f1a-fd857e002fbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"
Jul 24 18:16:10.698123 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.698095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvl5b\" (UniqueName: \"kubernetes.io/projected/024fd37c-99a2-4edc-9f1a-fd857e002fbd-kube-api-access-qvl5b\") pod \"payload-processing-e2e-inf-b-12361c-5889644b5b-hq998\" (UID: \"024fd37c-99a2-4edc-9f1a-fd857e002fbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"
Jul 24 18:16:10.698732 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.698710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d342795-84fc-4a18-9c69-efe23fb8dbcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22\" (UID: \"0d342795-84fc-4a18-9c69-efe23fb8dbcd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"
Jul 24 18:16:10.698775 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.698718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024fd37c-99a2-4edc-9f1a-fd857e002fbd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-12361c-5889644b5b-hq998\" (UID: \"024fd37c-99a2-4edc-9f1a-fd857e002fbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"
Jul 24 18:16:10.706595 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.706540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45j8p\" (UniqueName: \"kubernetes.io/projected/0d342795-84fc-4a18-9c69-efe23fb8dbcd-kube-api-access-45j8p\") pod \"payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22\" (UID: \"0d342795-84fc-4a18-9c69-efe23fb8dbcd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"
Jul 24 18:16:10.706712 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.706631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvl5b\" (UniqueName: \"kubernetes.io/projected/024fd37c-99a2-4edc-9f1a-fd857e002fbd-kube-api-access-qvl5b\") pod \"payload-processing-e2e-inf-b-12361c-5889644b5b-hq998\" (UID: \"024fd37c-99a2-4edc-9f1a-fd857e002fbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"
Jul 24 18:16:10.849079 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.849051    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"
Jul 24 18:16:10.881017 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:10.880988    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"
Jul 24 18:16:11.019480 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:11.019430    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"]
Jul 24 18:16:11.028193 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:16:11.028148    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d342795_84fc_4a18_9c69_efe23fb8dbcd.slice/crio-ea6b3401d37261712fc0b092e5c88c2898bc850c77fd514fba05dfa358949af1 WatchSource:0}: Error finding container ea6b3401d37261712fc0b092e5c88c2898bc850c77fd514fba05dfa358949af1: Status 404 returned error can't find the container with id ea6b3401d37261712fc0b092e5c88c2898bc850c77fd514fba05dfa358949af1
Jul 24 18:16:11.245527 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:11.245487    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"]
Jul 24 18:16:11.330723 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:11.330696    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:16:11.353009 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:11.352969    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22" event={"ID":"0d342795-84fc-4a18-9c69-efe23fb8dbcd","Type":"ContainerStarted","Data":"c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e"}
Jul 24 18:16:11.353177 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:11.353035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22" event={"ID":"0d342795-84fc-4a18-9c69-efe23fb8dbcd","Type":"ContainerStarted","Data":"ea6b3401d37261712fc0b092e5c88c2898bc850c77fd514fba05dfa358949af1"}
Jul 24 18:16:11.353831 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:11.353802    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"
Jul 24 18:16:11.356648 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:11.356612    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998" event={"ID":"024fd37c-99a2-4edc-9f1a-fd857e002fbd","Type":"ContainerStarted","Data":"6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8"}
Jul 24 18:16:11.356784 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:11.356674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998" event={"ID":"024fd37c-99a2-4edc-9f1a-fd857e002fbd","Type":"ContainerStarted","Data":"f341ffacba8d1ffdb6d6ebdbe2eddbf043d281fcd93b4ddcab5696d95a7baa38"}
Jul 24 18:16:11.356858 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:11.356791    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"
Jul 24 18:16:11.379649 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:11.378717    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22" podStartSLOduration=1.378693637 podStartE2EDuration="1.378693637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:16:11.376002678 +0000 UTC m=+2384.892513716" watchObservedRunningTime="2026-07-24 18:16:11.378693637 +0000 UTC m=+2384.895204675"
Jul 24 18:16:11.400276 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:11.400179    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998" podStartSLOduration=1.400157263 podStartE2EDuration="1.400157263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:16:11.397677585 +0000 UTC m=+2384.914188888" watchObservedRunningTime="2026-07-24 18:16:11.400157263 +0000 UTC m=+2384.916668297"
Jul 24 18:16:13.893996 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:13.893962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:14.316317 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:14.316291    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"
Jul 24 18:16:14.316469 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:14.316355    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"
Jul 24 18:16:16.331724 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:16.331683    2581 patch_prober.go:28] interesting pod/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 24 18:16:16.332131 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:16.331749    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 18:16:18.401099 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:18.401068    2581 generic.go:356] "Generic (PLEG): container finished" podID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerID="c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331" exitCode=0
Jul 24 18:16:18.401516 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:18.401138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" event={"ID":"6152dbd8-b79e-4f90-88d4-a298a5772a3f","Type":"ContainerDied","Data":"c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331"}
Jul 24 18:16:18.401516 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:18.401512    2581 scope.go:117] "RemoveContainer" containerID="c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331"
Jul 24 18:16:18.402528 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:18.402499    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:16:18.402622 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:18.402604    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:16:18.402668 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:18.402647    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:16:19.407133 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:19.407099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" event={"ID":"6152dbd8-b79e-4f90-88d4-a298a5772a3f","Type":"ContainerStarted","Data":"549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0"}
Jul 24 18:16:20.037313 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:20.037274    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:16:20.037532 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:20.037506    2581 patch_prober.go:28] interesting pod/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 24 18:16:20.037624 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:20.037605    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 24 18:16:20.482265 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:20.482232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:16:21.037352 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:21.037307    2581 patch_prober.go:28] interesting pod/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 24 18:16:21.037532 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:21.037369    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 24 18:16:22.037969 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:22.037931    2581 patch_prober.go:28] interesting pod/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 24 18:16:22.038333 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:22.038001    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 24 18:16:22.373759 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:22.373728    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"
Jul 24 18:16:23.037246 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:23.037210    2581 patch_prober.go:28] interesting pod/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 24 18:16:23.037420 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:23.037262    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 24 18:16:23.376396 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:23.376371    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"
Jul 24 18:16:24.041929 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:24.041896    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:16:24.043989 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:24.042245    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:16:24.043989 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:24.042854    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:16:24.340126 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:24.340086    2581 patch_prober.go:28] interesting pod/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 24 18:16:24.340291 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:24.340152    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 24 18:16:24.432823 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:24.432789    2581 generic.go:356] "Generic (PLEG): container finished" podID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerID="ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa" exitCode=0
Jul 24 18:16:24.433230 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:24.432865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" event={"ID":"34ea2cb7-17a3-454d-b837-40c6484cc2fa","Type":"ContainerDied","Data":"ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa"}
Jul 24 18:16:24.433396 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:24.433380    2581 scope.go:117] "RemoveContainer" containerID="ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa"
Jul 24 18:16:24.434358 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:24.434333    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:16:24.434446 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:24.434397    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:16:24.434446 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:24.434427    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:16:25.439280 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:25.439243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" event={"ID":"34ea2cb7-17a3-454d-b837-40c6484cc2fa","Type":"ContainerStarted","Data":"fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8"}
Jul 24 18:16:25.824707 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:25.824675    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:25.824944 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:25.824915    2581 patch_prober.go:28] interesting pod/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 24 18:16:25.825020 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:25.824975    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 24 18:16:26.824186 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:26.824150    2581 patch_prober.go:28] interesting pod/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 24 18:16:26.824546 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:26.824197    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 24 18:16:27.095244 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:27.095164    2581 scope.go:117] "RemoveContainer" containerID="bd34ccfdd66cb3dd596ac88e38fd0a6631b31a3b0e0203078bac14635cc5cd07"
Jul 24 18:16:27.825007 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:27.824973    2581 patch_prober.go:28] interesting pod/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 24 18:16:27.825400 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:27.825026    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 24 18:16:28.824507 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:28.824470    2581 patch_prober.go:28] interesting pod/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 24 18:16:28.824742 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:28.824523    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 24 18:16:29.828088 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:29.828059    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:29.828461 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:29.828404    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:29.828967 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:29.828949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:16:46.302500 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:16:46.302460    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:33.718902 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:33.718866    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:34.754408 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:34.754376    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:45.287856 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.287822    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"]
Jul 24 18:17:45.288314 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.288062    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5" podUID="765a4862-2f9c-453f-b19e-387804a60459" containerName="maas-api" containerID="cri-o://89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375" gracePeriod=30
Jul 24 18:17:45.318717 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.318690    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:45.376850 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.376820    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"]
Jul 24 18:17:45.377039 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.377014    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs" podUID="8247fad9-1f8f-4005-9208-20b97fcad722" containerName="payload-processing" containerID="cri-o://1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a" gracePeriod=30
Jul 24 18:17:45.385749 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.385725    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"]
Jul 24 18:17:45.385970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.385939    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp" podUID="ac324830-205b-4fb6-8864-64c178853ccb" containerName="payload-pre-processing" containerID="cri-o://8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05" gracePeriod=30
Jul 24 18:17:45.420623 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.420563    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:45.487889 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.487863    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:45.634612 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.634584    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:17:45.679085 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.679055    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"
Jul 24 18:17:45.695015 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.694992    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"
Jul 24 18:17:45.719399 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.719379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac324830-205b-4fb6-8864-64c178853ccb-plugins-config-volume\") pod \"ac324830-205b-4fb6-8864-64c178853ccb\" (UID: \"ac324830-205b-4fb6-8864-64c178853ccb\") "
Jul 24 18:17:45.719515 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.719425    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765a4862-2f9c-453f-b19e-387804a60459-openshift-service-ca\") pod \"765a4862-2f9c-453f-b19e-387804a60459\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") "
Jul 24 18:17:45.719515 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.719449    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk9sv\" (UniqueName: \"kubernetes.io/projected/ac324830-205b-4fb6-8864-64c178853ccb-kube-api-access-kk9sv\") pod \"ac324830-205b-4fb6-8864-64c178853ccb\" (UID: \"ac324830-205b-4fb6-8864-64c178853ccb\") "
Jul 24 18:17:45.719515 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.719485    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pktn6\" (UniqueName: \"kubernetes.io/projected/8247fad9-1f8f-4005-9208-20b97fcad722-kube-api-access-pktn6\") pod \"8247fad9-1f8f-4005-9208-20b97fcad722\" (UID: \"8247fad9-1f8f-4005-9208-20b97fcad722\") "
Jul 24 18:17:45.719695 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.719523    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlwhp\" (UniqueName: \"kubernetes.io/projected/765a4862-2f9c-453f-b19e-387804a60459-kube-api-access-jlwhp\") pod \"765a4862-2f9c-453f-b19e-387804a60459\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") "
Jul 24 18:17:45.719695 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.719557    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765a4862-2f9c-453f-b19e-387804a60459-maas-api-tls\") pod \"765a4862-2f9c-453f-b19e-387804a60459\" (UID: \"765a4862-2f9c-453f-b19e-387804a60459\") "
Jul 24 18:17:45.719695 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.719639    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8247fad9-1f8f-4005-9208-20b97fcad722-plugins-config-volume\") pod \"8247fad9-1f8f-4005-9208-20b97fcad722\" (UID: \"8247fad9-1f8f-4005-9208-20b97fcad722\") "
Jul 24 18:17:45.719916 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.719830    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac324830-205b-4fb6-8864-64c178853ccb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac324830-205b-4fb6-8864-64c178853ccb" (UID: "ac324830-205b-4fb6-8864-64c178853ccb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:17:45.719916 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.719826    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/765a4862-2f9c-453f-b19e-387804a60459-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "765a4862-2f9c-453f-b19e-387804a60459" (UID: "765a4862-2f9c-453f-b19e-387804a60459"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:17:45.720019 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.719942    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac324830-205b-4fb6-8864-64c178853ccb-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:17:45.720019 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.719961    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765a4862-2f9c-453f-b19e-387804a60459-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:17:45.720176 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.720149    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8247fad9-1f8f-4005-9208-20b97fcad722-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8247fad9-1f8f-4005-9208-20b97fcad722" (UID: "8247fad9-1f8f-4005-9208-20b97fcad722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:17:45.721855 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.721818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/765a4862-2f9c-453f-b19e-387804a60459-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "765a4862-2f9c-453f-b19e-387804a60459" (UID: "765a4862-2f9c-453f-b19e-387804a60459"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:17:45.721947 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.721887    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8247fad9-1f8f-4005-9208-20b97fcad722-kube-api-access-pktn6" (OuterVolumeSpecName: "kube-api-access-pktn6") pod "8247fad9-1f8f-4005-9208-20b97fcad722" (UID: "8247fad9-1f8f-4005-9208-20b97fcad722"). InnerVolumeSpecName "kube-api-access-pktn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:17:45.722043 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.722023    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac324830-205b-4fb6-8864-64c178853ccb-kube-api-access-kk9sv" (OuterVolumeSpecName: "kube-api-access-kk9sv") pod "ac324830-205b-4fb6-8864-64c178853ccb" (UID: "ac324830-205b-4fb6-8864-64c178853ccb"). InnerVolumeSpecName "kube-api-access-kk9sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:17:45.722097 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.722057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765a4862-2f9c-453f-b19e-387804a60459-kube-api-access-jlwhp" (OuterVolumeSpecName: "kube-api-access-jlwhp") pod "765a4862-2f9c-453f-b19e-387804a60459" (UID: "765a4862-2f9c-453f-b19e-387804a60459"). InnerVolumeSpecName "kube-api-access-jlwhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:17:45.806663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.806562    2581 generic.go:356] "Generic (PLEG): container finished" podID="765a4862-2f9c-453f-b19e-387804a60459" containerID="89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375" exitCode=0
Jul 24 18:17:45.806663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.806599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5" event={"ID":"765a4862-2f9c-453f-b19e-387804a60459","Type":"ContainerDied","Data":"89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375"}
Jul 24 18:17:45.806663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.806613    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"
Jul 24 18:17:45.806663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.806654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5" event={"ID":"765a4862-2f9c-453f-b19e-387804a60459","Type":"ContainerDied","Data":"61d28e21fc3a8e436c2f5d4282204a49d492ac16a8c85fbe93a5c9be9b8221f1"}
Jul 24 18:17:45.806961 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.806678    2581 scope.go:117] "RemoveContainer" containerID="89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375"
Jul 24 18:17:45.808118 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.808092    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac324830-205b-4fb6-8864-64c178853ccb" containerID="8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05" exitCode=0
Jul 24 18:17:45.808244 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.808138    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"
Jul 24 18:17:45.808244 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.808180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp" event={"ID":"ac324830-205b-4fb6-8864-64c178853ccb","Type":"ContainerDied","Data":"8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05"}
Jul 24 18:17:45.808244 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.808233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp" event={"ID":"ac324830-205b-4fb6-8864-64c178853ccb","Type":"ContainerDied","Data":"5eacdc5a2dc40953f2fd1151e30e9507bb7fa02b9a992a277c4deaedb8d5bd9c"}
Jul 24 18:17:45.809631 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.809607    2581 generic.go:356] "Generic (PLEG): container finished" podID="8247fad9-1f8f-4005-9208-20b97fcad722" containerID="1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a" exitCode=0
Jul 24 18:17:45.809759 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.809665    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"
Jul 24 18:17:45.809759 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.809669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs" event={"ID":"8247fad9-1f8f-4005-9208-20b97fcad722","Type":"ContainerDied","Data":"1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a"}
Jul 24 18:17:45.809873 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.809772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs" event={"ID":"8247fad9-1f8f-4005-9208-20b97fcad722","Type":"ContainerDied","Data":"347c99c5b9e13e04885ebe6b1256b721118f2cbf5acad528ed32d9bf95f94258"}
Jul 24 18:17:45.815887 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.815868    2581 scope.go:117] "RemoveContainer" containerID="89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375"
Jul 24 18:17:45.816222 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:17:45.816198    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375\": container with ID starting with 89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375 not found: ID does not exist" containerID="89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375"
Jul 24 18:17:45.816337 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.816233    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375"} err="failed to get container status \"89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375\": rpc error: code = NotFound desc = could not find container \"89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375\": container with ID starting with 89c05c6ffa8de806bec2d15e30a42f02ae43ad0970d19071eee078563fa00375 not found: ID does not exist"
Jul 24 18:17:45.816337 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.816257    2581 scope.go:117] "RemoveContainer" containerID="8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05"
Jul 24 18:17:45.820458 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.820439    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8247fad9-1f8f-4005-9208-20b97fcad722-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:17:45.820544 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.820461    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk9sv\" (UniqueName: \"kubernetes.io/projected/ac324830-205b-4fb6-8864-64c178853ccb-kube-api-access-kk9sv\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:17:45.820544 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.820475    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pktn6\" (UniqueName: \"kubernetes.io/projected/8247fad9-1f8f-4005-9208-20b97fcad722-kube-api-access-pktn6\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:17:45.820544 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.820491    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlwhp\" (UniqueName: \"kubernetes.io/projected/765a4862-2f9c-453f-b19e-387804a60459-kube-api-access-jlwhp\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:17:45.820544 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.820506    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765a4862-2f9c-453f-b19e-387804a60459-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:17:45.827019 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.827000    2581 scope.go:117] "RemoveContainer" containerID="8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05"
Jul 24 18:17:45.827321 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:17:45.827302    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05\": container with ID starting with 8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05 not found: ID does not exist" containerID="8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05"
Jul 24 18:17:45.827389 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.827327    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05"} err="failed to get container status \"8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05\": rpc error: code = NotFound desc = could not find container \"8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05\": container with ID starting with 8cc7bbcab6027963c7de2175ace0ee3b80a5983082a618786db08ca014cf9b05 not found: ID does not exist"
Jul 24 18:17:45.827389 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.827344    2581 scope.go:117] "RemoveContainer" containerID="1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a"
Jul 24 18:17:45.834674 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.834658    2581 scope.go:117] "RemoveContainer" containerID="1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a"
Jul 24 18:17:45.834908 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:17:45.834891    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a\": container with ID starting with 1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a not found: ID does not exist" containerID="1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a"
Jul 24 18:17:45.834961 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.834916    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a"} err="failed to get container status \"1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a\": rpc error: code = NotFound desc = could not find container \"1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a\": container with ID starting with 1473d9b743dcd9b1c86a8ed0e0db58db3af2c92690cbf4646ed59c04662bb41a not found: ID does not exist"
Jul 24 18:17:45.854729 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.854708    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"]
Jul 24 18:17:45.865329 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.865310    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3ffbc-6f4df74b55-dvkk5"]
Jul 24 18:17:45.882140 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.882118    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"]
Jul 24 18:17:45.892820 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.892784    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3ffbc-5c7d75786b-6ktjs"]
Jul 24 18:17:45.909136 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.909106    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"]
Jul 24 18:17:45.919004 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:45.918977    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3ffbc-6d4f757688-8rblp"]
Jul 24 18:17:46.958489 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:46.958457    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="765a4862-2f9c-453f-b19e-387804a60459" path="/var/lib/kubelet/pods/765a4862-2f9c-453f-b19e-387804a60459/volumes"
Jul 24 18:17:46.958878 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:46.958816    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8247fad9-1f8f-4005-9208-20b97fcad722" path="/var/lib/kubelet/pods/8247fad9-1f8f-4005-9208-20b97fcad722/volumes"
Jul 24 18:17:46.959123 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:46.959110    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac324830-205b-4fb6-8864-64c178853ccb" path="/var/lib/kubelet/pods/ac324830-205b-4fb6-8864-64c178853ccb/volumes"
Jul 24 18:17:49.906844 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:49.906811    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:50.329087 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:50.329059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:55.129945 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:55.129896    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:55.183766 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:55.183738    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:56.304585 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:56.304535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:56.375715 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:56.375685    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:56.559214 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:56.559180    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"]
Jul 24 18:17:56.559412 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:56.559388    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerName="istio-proxy" containerID="cri-o://549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0" gracePeriod=30
Jul 24 18:17:56.581943 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:56.581908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:17:56.825444 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:17:56.825357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:01.801153 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.801127    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:18:01.851006 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.850934    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istiod-ca-cert\") pod \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") "
Jul 24 18:18:01.851006 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.850982    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-workload-socket\") pod \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") "
Jul 24 18:18:01.851006 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.850999    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-data\") pod \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") "
Jul 24 18:18:01.851271 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851023    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-podinfo\") pod \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") "
Jul 24 18:18:01.851271 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851136    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgr8b\" (UniqueName: \"kubernetes.io/projected/6152dbd8-b79e-4f90-88d4-a298a5772a3f-kube-api-access-cgr8b\") pod \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") "
Jul 24 18:18:01.851271 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851197    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-envoy\") pod \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") "
Jul 24 18:18:01.851271 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851230    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-token\") pod \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") "
Jul 24 18:18:01.851271 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851254    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-credential-socket\") pod \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") "
Jul 24 18:18:01.851535 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851283    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-workload-certs\") pod \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\" (UID: \"6152dbd8-b79e-4f90-88d4-a298a5772a3f\") "
Jul 24 18:18:01.851535 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851283    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6152dbd8-b79e-4f90-88d4-a298a5772a3f" (UID: "6152dbd8-b79e-4f90-88d4-a298a5772a3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:18:01.851535 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851358    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6152dbd8-b79e-4f90-88d4-a298a5772a3f" (UID: "6152dbd8-b79e-4f90-88d4-a298a5772a3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:18:01.851535 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851482    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6152dbd8-b79e-4f90-88d4-a298a5772a3f" (UID: "6152dbd8-b79e-4f90-88d4-a298a5772a3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:18:01.851783 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851611    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:01.851783 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851630    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:01.851783 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851643    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:01.851783 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.851675    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6152dbd8-b79e-4f90-88d4-a298a5772a3f" (UID: "6152dbd8-b79e-4f90-88d4-a298a5772a3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:18:01.852065 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.852020    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-data" (OuterVolumeSpecName: "istio-data") pod "6152dbd8-b79e-4f90-88d4-a298a5772a3f" (UID: "6152dbd8-b79e-4f90-88d4-a298a5772a3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:18:01.853399 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.853373    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "6152dbd8-b79e-4f90-88d4-a298a5772a3f" (UID: "6152dbd8-b79e-4f90-88d4-a298a5772a3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:18:01.853399 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.853388    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6152dbd8-b79e-4f90-88d4-a298a5772a3f" (UID: "6152dbd8-b79e-4f90-88d4-a298a5772a3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:18:01.853399 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.853374    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6152dbd8-b79e-4f90-88d4-a298a5772a3f" (UID: "6152dbd8-b79e-4f90-88d4-a298a5772a3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 18:18:01.853556 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.853379    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6152dbd8-b79e-4f90-88d4-a298a5772a3f-kube-api-access-cgr8b" (OuterVolumeSpecName: "kube-api-access-cgr8b") pod "6152dbd8-b79e-4f90-88d4-a298a5772a3f" (UID: "6152dbd8-b79e-4f90-88d4-a298a5772a3f"). InnerVolumeSpecName "kube-api-access-cgr8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:18:01.883317 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.883292    2581 generic.go:356] "Generic (PLEG): container finished" podID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerID="549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0" exitCode=0
Jul 24 18:18:01.883409 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.883341    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"
Jul 24 18:18:01.883409 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.883358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" event={"ID":"6152dbd8-b79e-4f90-88d4-a298a5772a3f","Type":"ContainerDied","Data":"549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0"}
Jul 24 18:18:01.883409 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.883400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk" event={"ID":"6152dbd8-b79e-4f90-88d4-a298a5772a3f","Type":"ContainerDied","Data":"f7d694c8c578051374c930adf346a068472fbe4c7f9366858e913f2dcd7a716e"}
Jul 24 18:18:01.883524 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.883421    2581 scope.go:117] "RemoveContainer" containerID="549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0"
Jul 24 18:18:01.892449 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.892432    2581 scope.go:117] "RemoveContainer" containerID="c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331"
Jul 24 18:18:01.900170 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.900155    2581 scope.go:117] "RemoveContainer" containerID="549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0"
Jul 24 18:18:01.900431 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:18:01.900414    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0\": container with ID starting with 549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0 not found: ID does not exist" containerID="549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0"
Jul 24 18:18:01.900486 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.900440    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0"} err="failed to get container status \"549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0\": rpc error: code = NotFound desc = could not find container \"549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0\": container with ID starting with 549786e5183304a49157a4ad97fbc6456b0af9eca76739e8fbebbc28782c78c0 not found: ID does not exist"
Jul 24 18:18:01.900486 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.900460    2581 scope.go:117] "RemoveContainer" containerID="c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331"
Jul 24 18:18:01.900686 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:18:01.900671    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331\": container with ID starting with c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331 not found: ID does not exist" containerID="c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331"
Jul 24 18:18:01.900736 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.900690    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331"} err="failed to get container status \"c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331\": rpc error: code = NotFound desc = could not find container \"c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331\": container with ID starting with c89f9806a18a98b1a966fd96c308d4125e6431aa15414219c296b01f24a1d331 not found: ID does not exist"
Jul 24 18:18:01.905500 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.905478    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"]
Jul 24 18:18:01.911211 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.911191    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b3ffbc-openshift-default-846878d578-fqczk"]
Jul 24 18:18:01.953004 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.952984    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:01.953107 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.953007    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:01.953107 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.953017    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgr8b\" (UniqueName: \"kubernetes.io/projected/6152dbd8-b79e-4f90-88d4-a298a5772a3f-kube-api-access-cgr8b\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:01.953107 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.953025    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:01.953107 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.953035    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6152dbd8-b79e-4f90-88d4-a298a5772a3f-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:01.953107 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:01.953042    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6152dbd8-b79e-4f90-88d4-a298a5772a3f-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:02.957480 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:02.957452    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" path="/var/lib/kubelet/pods/6152dbd8-b79e-4f90-88d4-a298a5772a3f/volumes"
Jul 24 18:18:07.533078 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.533043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:07.557558 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.557520    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"]
Jul 24 18:18:07.557845 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.557790    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998" podUID="024fd37c-99a2-4edc-9f1a-fd857e002fbd" containerName="payload-processing" containerID="cri-o://6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8" gracePeriod=30
Jul 24 18:18:07.567006 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.566979    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"]
Jul 24 18:18:07.567184 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.567164    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22" podUID="0d342795-84fc-4a18-9c69-efe23fb8dbcd" containerName="payload-pre-processing" containerID="cri-o://c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e" gracePeriod=30
Jul 24 18:18:07.633809 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.633782    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:07.732063 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.732027    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:07.825390 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.825364    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"
Jul 24 18:18:07.846286 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.846263    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"
Jul 24 18:18:07.902959 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.902930    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45j8p\" (UniqueName: \"kubernetes.io/projected/0d342795-84fc-4a18-9c69-efe23fb8dbcd-kube-api-access-45j8p\") pod \"0d342795-84fc-4a18-9c69-efe23fb8dbcd\" (UID: \"0d342795-84fc-4a18-9c69-efe23fb8dbcd\") "
Jul 24 18:18:07.903114 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.902996    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d342795-84fc-4a18-9c69-efe23fb8dbcd-plugins-config-volume\") pod \"0d342795-84fc-4a18-9c69-efe23fb8dbcd\" (UID: \"0d342795-84fc-4a18-9c69-efe23fb8dbcd\") "
Jul 24 18:18:07.903114 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.903047    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024fd37c-99a2-4edc-9f1a-fd857e002fbd-plugins-config-volume\") pod \"024fd37c-99a2-4edc-9f1a-fd857e002fbd\" (UID: \"024fd37c-99a2-4edc-9f1a-fd857e002fbd\") "
Jul 24 18:18:07.903114 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.903075    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvl5b\" (UniqueName: \"kubernetes.io/projected/024fd37c-99a2-4edc-9f1a-fd857e002fbd-kube-api-access-qvl5b\") pod \"024fd37c-99a2-4edc-9f1a-fd857e002fbd\" (UID: \"024fd37c-99a2-4edc-9f1a-fd857e002fbd\") "
Jul 24 18:18:07.903384 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.903350    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d342795-84fc-4a18-9c69-efe23fb8dbcd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d342795-84fc-4a18-9c69-efe23fb8dbcd" (UID: "0d342795-84fc-4a18-9c69-efe23fb8dbcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:18:07.903509 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.903384    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024fd37c-99a2-4edc-9f1a-fd857e002fbd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "024fd37c-99a2-4edc-9f1a-fd857e002fbd" (UID: "024fd37c-99a2-4edc-9f1a-fd857e002fbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:18:07.905027 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.905006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024fd37c-99a2-4edc-9f1a-fd857e002fbd-kube-api-access-qvl5b" (OuterVolumeSpecName: "kube-api-access-qvl5b") pod "024fd37c-99a2-4edc-9f1a-fd857e002fbd" (UID: "024fd37c-99a2-4edc-9f1a-fd857e002fbd"). InnerVolumeSpecName "kube-api-access-qvl5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:18:07.905109 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.905040    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d342795-84fc-4a18-9c69-efe23fb8dbcd-kube-api-access-45j8p" (OuterVolumeSpecName: "kube-api-access-45j8p") pod "0d342795-84fc-4a18-9c69-efe23fb8dbcd" (UID: "0d342795-84fc-4a18-9c69-efe23fb8dbcd"). InnerVolumeSpecName "kube-api-access-45j8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:18:07.910552 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.910531    2581 generic.go:356] "Generic (PLEG): container finished" podID="0d342795-84fc-4a18-9c69-efe23fb8dbcd" containerID="c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e" exitCode=0
Jul 24 18:18:07.910660 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.910593    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"
Jul 24 18:18:07.910660 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.910614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22" event={"ID":"0d342795-84fc-4a18-9c69-efe23fb8dbcd","Type":"ContainerDied","Data":"c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e"}
Jul 24 18:18:07.910751 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.910666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22" event={"ID":"0d342795-84fc-4a18-9c69-efe23fb8dbcd","Type":"ContainerDied","Data":"ea6b3401d37261712fc0b092e5c88c2898bc850c77fd514fba05dfa358949af1"}
Jul 24 18:18:07.910751 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.910697    2581 scope.go:117] "RemoveContainer" containerID="c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e"
Jul 24 18:18:07.911892 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.911871    2581 generic.go:356] "Generic (PLEG): container finished" podID="024fd37c-99a2-4edc-9f1a-fd857e002fbd" containerID="6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8" exitCode=0
Jul 24 18:18:07.911995 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.911916    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"
Jul 24 18:18:07.911995 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.911942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998" event={"ID":"024fd37c-99a2-4edc-9f1a-fd857e002fbd","Type":"ContainerDied","Data":"6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8"}
Jul 24 18:18:07.911995 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.911978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998" event={"ID":"024fd37c-99a2-4edc-9f1a-fd857e002fbd","Type":"ContainerDied","Data":"f341ffacba8d1ffdb6d6ebdbe2eddbf043d281fcd93b4ddcab5696d95a7baa38"}
Jul 24 18:18:07.918845 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.918828    2581 scope.go:117] "RemoveContainer" containerID="c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e"
Jul 24 18:18:07.919106 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:18:07.919089    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e\": container with ID starting with c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e not found: ID does not exist" containerID="c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e"
Jul 24 18:18:07.919162 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.919114    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e"} err="failed to get container status \"c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e\": rpc error: code = NotFound desc = could not find container \"c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e\": container with ID starting with c09bb31d8a6d0751b3a1ce53d4e8a165e686443f5c8d764fe1c6f73dccf5b86e not found: ID does not exist"
Jul 24 18:18:07.919162 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.919133    2581 scope.go:117] "RemoveContainer" containerID="6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8"
Jul 24 18:18:07.926473 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.926458    2581 scope.go:117] "RemoveContainer" containerID="6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8"
Jul 24 18:18:07.926741 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:18:07.926720    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8\": container with ID starting with 6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8 not found: ID does not exist" containerID="6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8"
Jul 24 18:18:07.926808 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.926749    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8"} err="failed to get container status \"6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8\": rpc error: code = NotFound desc = could not find container \"6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8\": container with ID starting with 6a5d83ddf1218a4b00b6976c2291967df1a9fa247936d72da01b50568b2133c8 not found: ID does not exist"
Jul 24 18:18:07.935199 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.935178    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"]
Jul 24 18:18:07.938430 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.938410    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12361c-5889644b5b-hq998"]
Jul 24 18:18:07.948425 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.948402    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"]
Jul 24 18:18:07.950796 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:07.950777    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12361c-5b7979f5dd-c6x22"]
Jul 24 18:18:08.003800 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:08.003778    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45j8p\" (UniqueName: \"kubernetes.io/projected/0d342795-84fc-4a18-9c69-efe23fb8dbcd-kube-api-access-45j8p\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:08.003800 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:08.003801    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d342795-84fc-4a18-9c69-efe23fb8dbcd-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:08.003915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:08.003812    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024fd37c-99a2-4edc-9f1a-fd857e002fbd-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:08.003915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:08.003821    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvl5b\" (UniqueName: \"kubernetes.io/projected/024fd37c-99a2-4edc-9f1a-fd857e002fbd-kube-api-access-qvl5b\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:08.957486 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:08.957412    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="024fd37c-99a2-4edc-9f1a-fd857e002fbd" path="/var/lib/kubelet/pods/024fd37c-99a2-4edc-9f1a-fd857e002fbd/volumes"
Jul 24 18:18:08.957838 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:08.957780    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d342795-84fc-4a18-9c69-efe23fb8dbcd" path="/var/lib/kubelet/pods/0d342795-84fc-4a18-9c69-efe23fb8dbcd/volumes"
Jul 24 18:18:12.036710 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:12.036673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:12.416293 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:12.416261    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:18.201699 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:18.201667    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:18.271231 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:18.271202    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:18.458511 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:18.458429    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"]
Jul 24 18:18:18.458758 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:18.458724    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerName="istio-proxy" containerID="cri-o://fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8" gracePeriod=30
Jul 24 18:18:18.485592 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:18.485554    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:18.721343 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:18.721255    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:19.055968 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.055862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:19.357494 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357465    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"]
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357765    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerName="istio-proxy"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357777    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerName="istio-proxy"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357786    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d342795-84fc-4a18-9c69-efe23fb8dbcd" containerName="payload-pre-processing"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357791    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d342795-84fc-4a18-9c69-efe23fb8dbcd" containerName="payload-pre-processing"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357799    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8247fad9-1f8f-4005-9208-20b97fcad722" containerName="payload-processing"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357804    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8247fad9-1f8f-4005-9208-20b97fcad722" containerName="payload-processing"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357812    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerName="istio-proxy"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357817    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerName="istio-proxy"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357824    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024fd37c-99a2-4edc-9f1a-fd857e002fbd" containerName="payload-processing"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357830    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="024fd37c-99a2-4edc-9f1a-fd857e002fbd" containerName="payload-processing"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357842    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="765a4862-2f9c-453f-b19e-387804a60459" containerName="maas-api"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357847    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="765a4862-2f9c-453f-b19e-387804a60459" containerName="maas-api"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357856    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac324830-205b-4fb6-8864-64c178853ccb" containerName="payload-pre-processing"
Jul 24 18:18:19.357870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357861    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac324830-205b-4fb6-8864-64c178853ccb" containerName="payload-pre-processing"
Jul 24 18:18:19.358302 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357902    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="765a4862-2f9c-453f-b19e-387804a60459" containerName="maas-api"
Jul 24 18:18:19.358302 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357910    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerName="istio-proxy"
Jul 24 18:18:19.358302 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357916    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d342795-84fc-4a18-9c69-efe23fb8dbcd" containerName="payload-pre-processing"
Jul 24 18:18:19.358302 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357922    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8247fad9-1f8f-4005-9208-20b97fcad722" containerName="payload-processing"
Jul 24 18:18:19.358302 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357927    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac324830-205b-4fb6-8864-64c178853ccb" containerName="payload-pre-processing"
Jul 24 18:18:19.358302 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.357933    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="024fd37c-99a2-4edc-9f1a-fd857e002fbd" containerName="payload-processing"
Jul 24 18:18:19.360947 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.360931    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.363110 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.363089    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dcf34d-openshift-default-dockercfg-fcfvf\""
Jul 24 18:18:19.372273 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.372252    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"]
Jul 24 18:18:19.377030 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.377007    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:19.499490 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.499453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c882666b-2b38-4a49-9f83-9271e7a39681-istio-podinfo\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.499704 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.499497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-credential-socket\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.499704 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.499562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjxvf\" (UniqueName: \"kubernetes.io/projected/c882666b-2b38-4a49-9f83-9271e7a39681-kube-api-access-hjxvf\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.499704 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.499627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-istio-envoy\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.499704 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.499654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c882666b-2b38-4a49-9f83-9271e7a39681-istiod-ca-cert\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.499704 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.499684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-workload-certs\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.499908 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.499752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-istio-data\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.499908 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.499783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c882666b-2b38-4a49-9f83-9271e7a39681-istio-token\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.499908 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.499809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-workload-socket\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.600827 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.600791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c882666b-2b38-4a49-9f83-9271e7a39681-istio-podinfo\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.600827 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.600829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-credential-socket\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.601023 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.600853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjxvf\" (UniqueName: \"kubernetes.io/projected/c882666b-2b38-4a49-9f83-9271e7a39681-kube-api-access-hjxvf\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.601023 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.600889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-istio-envoy\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.601023 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.600906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c882666b-2b38-4a49-9f83-9271e7a39681-istiod-ca-cert\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.601023 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.600926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-workload-certs\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.601023 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.600961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-istio-data\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.601023 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.600987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c882666b-2b38-4a49-9f83-9271e7a39681-istio-token\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.601368 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.601071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-workload-socket\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.601368 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.601247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-credential-socket\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.601482 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.601447    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-istio-data\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.601482 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.601469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-workload-socket\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.601563 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.601514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-workload-certs\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.601679 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.601661    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c882666b-2b38-4a49-9f83-9271e7a39681-istiod-ca-cert\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.603205 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.603177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-istio-envoy\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.603407 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.603388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c882666b-2b38-4a49-9f83-9271e7a39681-istio-podinfo\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.609404 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.609351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c882666b-2b38-4a49-9f83-9271e7a39681-istio-token\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.609718 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.609698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjxvf\" (UniqueName: \"kubernetes.io/projected/c882666b-2b38-4a49-9f83-9271e7a39681-kube-api-access-hjxvf\") pod \"e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") " pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.670513 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.670491    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:19.802073 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.802037    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"]
Jul 24 18:18:19.804930 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:18:19.804901    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc882666b_2b38_4a49_9f83_9271e7a39681.slice/crio-f5da9de9e7c1d715a4ed0d8fdbe0a1d5c121167c6cf0234a6c1bb3de60859fbb WatchSource:0}: Error finding container f5da9de9e7c1d715a4ed0d8fdbe0a1d5c121167c6cf0234a6c1bb3de60859fbb: Status 404 returned error can't find the container with id f5da9de9e7c1d715a4ed0d8fdbe0a1d5c121167c6cf0234a6c1bb3de60859fbb
Jul 24 18:18:19.807210 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.807176    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:18:19.807280 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.807253    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:18:19.807325 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.807296    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:18:19.966412 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.966379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" event={"ID":"c882666b-2b38-4a49-9f83-9271e7a39681","Type":"ContainerStarted","Data":"5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884"}
Jul 24 18:18:19.966559 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.966427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" event={"ID":"c882666b-2b38-4a49-9f83-9271e7a39681","Type":"ContainerStarted","Data":"f5da9de9e7c1d715a4ed0d8fdbe0a1d5c121167c6cf0234a6c1bb3de60859fbb"}
Jul 24 18:18:19.987608 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:19.987537    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" podStartSLOduration=0.987520387 podStartE2EDuration="987.520387ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:18:19.985830764 +0000 UTC m=+2513.502341800" watchObservedRunningTime="2026-07-24 18:18:19.987520387 +0000 UTC m=+2513.504031422"
Jul 24 18:18:20.486858 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:20.486822    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:20.671389 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:20.671357    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:20.676736 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:20.676712    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:20.970645 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:20.970616    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:20.971474 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:20.971456    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:21.950237 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:21.950188    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:22.030377 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.030346    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"]
Jul 24 18:18:22.030734 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.030719    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6152dbd8-b79e-4f90-88d4-a298a5772a3f" containerName="istio-proxy"
Jul 24 18:18:22.033687 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.033670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:22.038393 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.038362    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:18:22.038526 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.038459    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:18:22.038616 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.038603    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dcf34d-serving-cert\""
Jul 24 18:18:22.039337 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.039318    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fvm85\""
Jul 24 18:18:22.060208 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.060181    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"]
Jul 24 18:18:22.066510 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.066484    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:22.123564 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.123529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfpjz\" (UniqueName: \"kubernetes.io/projected/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-kube-api-access-xfpjz\") pod \"maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b\" (UID: \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:22.123798 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.123702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-maas-api-tls\") pod \"maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b\" (UID: \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:22.123872 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.123825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-openshift-service-ca\") pod \"maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b\" (UID: \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:22.211298 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.211203    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:22.225341 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.225311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-openshift-service-ca\") pod \"maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b\" (UID: \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:22.225494 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.225382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfpjz\" (UniqueName: \"kubernetes.io/projected/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-kube-api-access-xfpjz\") pod \"maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b\" (UID: \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:22.225494 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.225414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-maas-api-tls\") pod \"maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b\" (UID: \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:22.225937 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.225917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-openshift-service-ca\") pod \"maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b\" (UID: \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:22.227779 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.227750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-maas-api-tls\") pod \"maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b\" (UID: \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:22.244552 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.244522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfpjz\" (UniqueName: \"kubernetes.io/projected/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-kube-api-access-xfpjz\") pod \"maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b\" (UID: \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:22.326717 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.326686    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:22.344088 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.344061    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:22.471508 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.471480    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"]
Jul 24 18:18:22.474045 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:18:22.474006    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9f71d9_6ca3_4c5e_8ddb_1eec243a8421.slice/crio-65b4bce308c25c4f991e52edb597f10c4539d5dda523c1f9907e16359b8b2851 WatchSource:0}: Error finding container 65b4bce308c25c4f991e52edb597f10c4539d5dda523c1f9907e16359b8b2851: Status 404 returned error can't find the container with id 65b4bce308c25c4f991e52edb597f10c4539d5dda523c1f9907e16359b8b2851
Jul 24 18:18:22.475964 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.475946    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:18:22.700684 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.700652    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:22.798288 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.798199    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:22.835358 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.835327    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"]
Jul 24 18:18:22.839704 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.839688    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"
Jul 24 18:18:22.841971 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.841951    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dcf34d-dockercfg-knnnk\""
Jul 24 18:18:22.842090 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.841951    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dcf34d\""
Jul 24 18:18:22.860587 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.851702    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"]
Jul 24 18:18:22.932724 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.932686    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9kn9\" (UniqueName: \"kubernetes.io/projected/e70a3f33-0181-441a-b298-edc10db3872f-kube-api-access-n9kn9\") pod \"payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b\" (UID: \"e70a3f33-0181-441a-b298-edc10db3872f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"
Jul 24 18:18:22.932908 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.932755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70a3f33-0181-441a-b298-edc10db3872f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b\" (UID: \"e70a3f33-0181-441a-b298-edc10db3872f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"
Jul 24 18:18:22.980815 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:22.980773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b" event={"ID":"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421","Type":"ContainerStarted","Data":"65b4bce308c25c4f991e52edb597f10c4539d5dda523c1f9907e16359b8b2851"}
Jul 24 18:18:23.033446 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.033421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9kn9\" (UniqueName: \"kubernetes.io/projected/e70a3f33-0181-441a-b298-edc10db3872f-kube-api-access-n9kn9\") pod \"payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b\" (UID: \"e70a3f33-0181-441a-b298-edc10db3872f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"
Jul 24 18:18:23.033565 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.033471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70a3f33-0181-441a-b298-edc10db3872f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b\" (UID: \"e70a3f33-0181-441a-b298-edc10db3872f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"
Jul 24 18:18:23.034398 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.034374    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70a3f33-0181-441a-b298-edc10db3872f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b\" (UID: \"e70a3f33-0181-441a-b298-edc10db3872f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"
Jul 24 18:18:23.042245 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.042223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9kn9\" (UniqueName: \"kubernetes.io/projected/e70a3f33-0181-441a-b298-edc10db3872f-kube-api-access-n9kn9\") pod \"payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b\" (UID: \"e70a3f33-0181-441a-b298-edc10db3872f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"
Jul 24 18:18:23.158928 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.158896    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"
Jul 24 18:18:23.495443 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.495364    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"]
Jul 24 18:18:23.537398 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:18:23.537354    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70a3f33_0181_441a_b298_edc10db3872f.slice/crio-68fd91737785e9b37a336bbc099809210347fb028fee0568ce3d85ea3185241b WatchSource:0}: Error finding container 68fd91737785e9b37a336bbc099809210347fb028fee0568ce3d85ea3185241b: Status 404 returned error can't find the container with id 68fd91737785e9b37a336bbc099809210347fb028fee0568ce3d85ea3185241b
Jul 24 18:18:23.721966 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.721911    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:18:23.839782 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.839746    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-workload-certs\") pod \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") "
Jul 24 18:18:23.839969 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.839800    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w72kv\" (UniqueName: \"kubernetes.io/projected/34ea2cb7-17a3-454d-b837-40c6484cc2fa-kube-api-access-w72kv\") pod \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") "
Jul 24 18:18:23.839969 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.839819    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-credential-socket\") pod \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") "
Jul 24 18:18:23.839969 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.839837    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-workload-socket\") pod \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") "
Jul 24 18:18:23.839969 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.839871    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-envoy\") pod \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") "
Jul 24 18:18:23.839969 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.839911    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-podinfo\") pod \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") "
Jul 24 18:18:23.839969 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.839947    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-data\") pod \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") "
Jul 24 18:18:23.840289 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.839992    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istiod-ca-cert\") pod \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") "
Jul 24 18:18:23.840289 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.840019    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-token\") pod \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\" (UID: \"34ea2cb7-17a3-454d-b837-40c6484cc2fa\") "
Jul 24 18:18:23.840289 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.840081    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34ea2cb7-17a3-454d-b837-40c6484cc2fa" (UID: "34ea2cb7-17a3-454d-b837-40c6484cc2fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:18:23.840289 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.840144    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34ea2cb7-17a3-454d-b837-40c6484cc2fa" (UID: "34ea2cb7-17a3-454d-b837-40c6484cc2fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:18:23.840289 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.840274    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:23.840515 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.840292    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:23.840515 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.840434    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34ea2cb7-17a3-454d-b837-40c6484cc2fa" (UID: "34ea2cb7-17a3-454d-b837-40c6484cc2fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:18:23.840515 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.840490    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34ea2cb7-17a3-454d-b837-40c6484cc2fa" (UID: "34ea2cb7-17a3-454d-b837-40c6484cc2fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:18:23.841180 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.841152    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "34ea2cb7-17a3-454d-b837-40c6484cc2fa" (UID: "34ea2cb7-17a3-454d-b837-40c6484cc2fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:18:23.842180 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.842156    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34ea2cb7-17a3-454d-b837-40c6484cc2fa" (UID: "34ea2cb7-17a3-454d-b837-40c6484cc2fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:18:23.842399 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.842380    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "34ea2cb7-17a3-454d-b837-40c6484cc2fa" (UID: "34ea2cb7-17a3-454d-b837-40c6484cc2fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:18:23.842502 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.842481    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34ea2cb7-17a3-454d-b837-40c6484cc2fa" (UID: "34ea2cb7-17a3-454d-b837-40c6484cc2fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 18:18:23.842554 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.842496    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ea2cb7-17a3-454d-b837-40c6484cc2fa-kube-api-access-w72kv" (OuterVolumeSpecName: "kube-api-access-w72kv") pod "34ea2cb7-17a3-454d-b837-40c6484cc2fa" (UID: "34ea2cb7-17a3-454d-b837-40c6484cc2fa"). InnerVolumeSpecName "kube-api-access-w72kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:18:23.941060 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.941031    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72kv\" (UniqueName: \"kubernetes.io/projected/34ea2cb7-17a3-454d-b837-40c6484cc2fa-kube-api-access-w72kv\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:23.941060 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.941059    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:23.941235 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.941072    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:23.941235 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.941085    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:23.941235 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.941095    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:23.941235 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.941105    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:23.941235 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.941116    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ea2cb7-17a3-454d-b837-40c6484cc2fa-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:23.987268 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.987182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b" event={"ID":"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421","Type":"ContainerStarted","Data":"3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2"}
Jul 24 18:18:23.987702 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.987280    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:23.988875 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.988851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b" event={"ID":"e70a3f33-0181-441a-b298-edc10db3872f","Type":"ContainerStarted","Data":"c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3"}
Jul 24 18:18:23.988970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.988902    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"
Jul 24 18:18:23.988970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.988928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b" event={"ID":"e70a3f33-0181-441a-b298-edc10db3872f","Type":"ContainerStarted","Data":"68fd91737785e9b37a336bbc099809210347fb028fee0568ce3d85ea3185241b"}
Jul 24 18:18:23.990454 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.990431    2581 generic.go:356] "Generic (PLEG): container finished" podID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerID="fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8" exitCode=0
Jul 24 18:18:23.990537 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.990481    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"
Jul 24 18:18:23.990537 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.990494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" event={"ID":"34ea2cb7-17a3-454d-b837-40c6484cc2fa","Type":"ContainerDied","Data":"fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8"}
Jul 24 18:18:23.990639 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.990534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg" event={"ID":"34ea2cb7-17a3-454d-b837-40c6484cc2fa","Type":"ContainerDied","Data":"b44962b35354c5eb94083e3b8603a9181add66215434d120139ad5620b443a2b"}
Jul 24 18:18:23.990639 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.990560    2581 scope.go:117] "RemoveContainer" containerID="fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8"
Jul 24 18:18:23.998966 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:23.998943    2581 scope.go:117] "RemoveContainer" containerID="ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa"
Jul 24 18:18:24.007044 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:24.007028    2581 scope.go:117] "RemoveContainer" containerID="fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8"
Jul 24 18:18:24.007316 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:18:24.007289    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8\": container with ID starting with fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8 not found: ID does not exist" containerID="fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8"
Jul 24 18:18:24.007401 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:24.007323    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8"} err="failed to get container status \"fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8\": rpc error: code = NotFound desc = could not find container \"fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8\": container with ID starting with fdcbf592d654fca03c2b550f25125c226919c7d0d05f4784982603995901f1b8 not found: ID does not exist"
Jul 24 18:18:24.007401 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:24.007342    2581 scope.go:117] "RemoveContainer" containerID="ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa"
Jul 24 18:18:24.007648 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:18:24.007604    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa\": container with ID starting with ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa not found: ID does not exist" containerID="ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa"
Jul 24 18:18:24.007736 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:24.007653    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa"} err="failed to get container status \"ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa\": rpc error: code = NotFound desc = could not find container \"ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa\": container with ID starting with ef4dacbac36a622784c29e5077bbdb75428634bdf528676e71eceaa388a9c5aa not found: ID does not exist"
Jul 24 18:18:24.008703 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:24.008657    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b" podStartSLOduration=1.487638473 podStartE2EDuration="2.008641466s" totalImagesPullingTime="521.002993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:18:24.008357794 +0000 UTC m=+2517.524868829" watchObservedRunningTime="2026-07-24 18:18:24.008641466 +0000 UTC m=+2517.525152507"
Jul 24 18:18:24.029305 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:24.029266    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b" podStartSLOduration=2.029253619 podStartE2EDuration="2.029253619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:18:24.027645487 +0000 UTC m=+2517.544156525" watchObservedRunningTime="2026-07-24 18:18:24.029253619 +0000 UTC m=+2517.545764654"
Jul 24 18:18:24.039852 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:24.039827    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"]
Jul 24 18:18:24.049234 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:24.049213    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-12361c-openshift-default-65df5c8b7b-9f8bg"]
Jul 24 18:18:24.957189 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:24.957157    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" path="/var/lib/kubelet/pods/34ea2cb7-17a3-454d-b837-40c6484cc2fa/volumes"
Jul 24 18:18:26.488130 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:26.488098    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:26.963488 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:26.963452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:27.248109 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.248021    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"]
Jul 24 18:18:27.248436 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.248418    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerName="istio-proxy"
Jul 24 18:18:27.248510 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.248439    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerName="istio-proxy"
Jul 24 18:18:27.248510 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.248471    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerName="istio-proxy"
Jul 24 18:18:27.248510 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.248479    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerName="istio-proxy"
Jul 24 18:18:27.248640 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.248557    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerName="istio-proxy"
Jul 24 18:18:27.248640 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.248587    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34ea2cb7-17a3-454d-b837-40c6484cc2fa" containerName="istio-proxy"
Jul 24 18:18:27.253177 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.253157    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.255994 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.255975    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4a14d5-openshift-default-dockercfg-sr7b9\""
Jul 24 18:18:27.264156 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.264138    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"]
Jul 24 18:18:27.284634 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.284596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:27.367305 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.367273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-token\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.367305 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.367308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-workload-socket\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.367537 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.367397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2cj\" (UniqueName: \"kubernetes.io/projected/cc04ac9a-531b-482b-af64-45306f05eb0a-kube-api-access-pn2cj\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.367537 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.367439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc04ac9a-531b-482b-af64-45306f05eb0a-istiod-ca-cert\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.367537 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.367475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-envoy\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.367537 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.367499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-podinfo\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.367706 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.367595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-data\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.367706 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.367624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-workload-certs\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.367706 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.367666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-credential-socket\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.468841 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.468809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-workload-certs\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.468841 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.468847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-credential-socket\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.469091 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.468887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-token\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.469091 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.468905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-workload-socket\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.469091 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.468927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2cj\" (UniqueName: \"kubernetes.io/projected/cc04ac9a-531b-482b-af64-45306f05eb0a-kube-api-access-pn2cj\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.469091 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.468944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc04ac9a-531b-482b-af64-45306f05eb0a-istiod-ca-cert\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.469091 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.468963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-envoy\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.469091 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.468989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-podinfo\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.469091 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.469024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-data\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.469453 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.469235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-workload-certs\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.469453 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.469310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-workload-socket\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.469587 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.469475    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-data\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.469653 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.469622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-credential-socket\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.469800 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.469781    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc04ac9a-531b-482b-af64-45306f05eb0a-istiod-ca-cert\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.471155 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.471133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-envoy\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.471553 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.471534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-podinfo\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.478939 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.478917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-token\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.479114 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.479098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2cj\" (UniqueName: \"kubernetes.io/projected/cc04ac9a-531b-482b-af64-45306f05eb0a-kube-api-access-pn2cj\") pod \"e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") " pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.564154 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.564130    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:27.693030 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.693005    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"]
Jul 24 18:18:27.694969 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:18:27.694943    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc04ac9a_531b_482b_af64_45306f05eb0a.slice/crio-e8294d741f1fbd5f919edcfb1772b27292bd2f1f0eeaa9aafa131df527314d09 WatchSource:0}: Error finding container e8294d741f1fbd5f919edcfb1772b27292bd2f1f0eeaa9aafa131df527314d09: Status 404 returned error can't find the container with id e8294d741f1fbd5f919edcfb1772b27292bd2f1f0eeaa9aafa131df527314d09
Jul 24 18:18:27.697162 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.697132    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:18:27.697222 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.697194    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:18:27.697261 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.697222    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:18:27.768676 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:27.768650    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:28.015391 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:28.015311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" event={"ID":"cc04ac9a-531b-482b-af64-45306f05eb0a","Type":"ContainerStarted","Data":"6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb"}
Jul 24 18:18:28.015391 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:28.015366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" event={"ID":"cc04ac9a-531b-482b-af64-45306f05eb0a","Type":"ContainerStarted","Data":"e8294d741f1fbd5f919edcfb1772b27292bd2f1f0eeaa9aafa131df527314d09"}
Jul 24 18:18:28.036427 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:28.036385    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" podStartSLOduration=1.036370694 podStartE2EDuration="1.036370694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:18:28.03551373 +0000 UTC m=+2521.552024767" watchObservedRunningTime="2026-07-24 18:18:28.036370694 +0000 UTC m=+2521.552881730"
Jul 24 18:18:28.565284 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:28.565253    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:28.570172 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:28.570147    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:29.019633 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:29.019542    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:29.020640 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:29.020622    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:29.961916 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:29.961880    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:30.001359 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.001334    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:30.058816 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.058787    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:30.059767 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.059738    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"]
Jul 24 18:18:30.063725 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.063703    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:30.066311 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.066292    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4a14d5-serving-cert\""
Jul 24 18:18:30.074606 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.074582    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"]
Jul 24 18:18:30.140043 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.140015    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:30.192232 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.192191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gczg6\" (UniqueName: \"kubernetes.io/projected/24afbe93-338c-4cb3-ae61-fb780feda33b-kube-api-access-gczg6\") pod \"maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8\" (UID: \"24afbe93-338c-4cb3-ae61-fb780feda33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:30.192405 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.192297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24afbe93-338c-4cb3-ae61-fb780feda33b-openshift-service-ca\") pod \"maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8\" (UID: \"24afbe93-338c-4cb3-ae61-fb780feda33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:30.192405 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.192324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24afbe93-338c-4cb3-ae61-fb780feda33b-maas-api-tls\") pod \"maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8\" (UID: \"24afbe93-338c-4cb3-ae61-fb780feda33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:30.244893 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.244806    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:30.292751 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.292719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24afbe93-338c-4cb3-ae61-fb780feda33b-openshift-service-ca\") pod \"maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8\" (UID: \"24afbe93-338c-4cb3-ae61-fb780feda33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:30.292751 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.292753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24afbe93-338c-4cb3-ae61-fb780feda33b-maas-api-tls\") pod \"maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8\" (UID: \"24afbe93-338c-4cb3-ae61-fb780feda33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:30.292924 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.292808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gczg6\" (UniqueName: \"kubernetes.io/projected/24afbe93-338c-4cb3-ae61-fb780feda33b-kube-api-access-gczg6\") pod \"maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8\" (UID: \"24afbe93-338c-4cb3-ae61-fb780feda33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:30.293421 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.293401    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24afbe93-338c-4cb3-ae61-fb780feda33b-openshift-service-ca\") pod \"maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8\" (UID: \"24afbe93-338c-4cb3-ae61-fb780feda33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:30.295334 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.295310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24afbe93-338c-4cb3-ae61-fb780feda33b-maas-api-tls\") pod \"maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8\" (UID: \"24afbe93-338c-4cb3-ae61-fb780feda33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:30.303301 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.303278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gczg6\" (UniqueName: \"kubernetes.io/projected/24afbe93-338c-4cb3-ae61-fb780feda33b-kube-api-access-gczg6\") pod \"maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8\" (UID: \"24afbe93-338c-4cb3-ae61-fb780feda33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:30.377135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.377111    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:30.721006 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.720969    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"]
Jul 24 18:18:30.724384 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:18:30.724356    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24afbe93_338c_4cb3_ae61_fb780feda33b.slice/crio-3aaeb02df1ed2307a1ce748f3cb8b28f99a3a035a837d3a3b629e93bc31de0b1 WatchSource:0}: Error finding container 3aaeb02df1ed2307a1ce748f3cb8b28f99a3a035a837d3a3b629e93bc31de0b1: Status 404 returned error can't find the container with id 3aaeb02df1ed2307a1ce748f3cb8b28f99a3a035a837d3a3b629e93bc31de0b1
Jul 24 18:18:30.810381 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.810347    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:30.906401 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:30.906361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:31.006382 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.006301    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"]
Jul 24 18:18:31.010685 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.010664    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"
Jul 24 18:18:31.014628 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.014601    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4a14d5\""
Jul 24 18:18:31.014732 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.014698    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4a14d5-dockercfg-9xkrq\""
Jul 24 18:18:31.024754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.024727    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"]
Jul 24 18:18:31.029828 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.029806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8" event={"ID":"24afbe93-338c-4cb3-ae61-fb780feda33b","Type":"ContainerStarted","Data":"3aaeb02df1ed2307a1ce748f3cb8b28f99a3a035a837d3a3b629e93bc31de0b1"}
Jul 24 18:18:31.104784 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.104756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5b2c47-1a97-4e84-9153-b23a84840b82-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v\" (UID: \"9d5b2c47-1a97-4e84-9153-b23a84840b82\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"
Jul 24 18:18:31.104939 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.104915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln4l6\" (UniqueName: \"kubernetes.io/projected/9d5b2c47-1a97-4e84-9153-b23a84840b82-kube-api-access-ln4l6\") pod \"payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v\" (UID: \"9d5b2c47-1a97-4e84-9153-b23a84840b82\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"
Jul 24 18:18:31.206236 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.206202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln4l6\" (UniqueName: \"kubernetes.io/projected/9d5b2c47-1a97-4e84-9153-b23a84840b82-kube-api-access-ln4l6\") pod \"payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v\" (UID: \"9d5b2c47-1a97-4e84-9153-b23a84840b82\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"
Jul 24 18:18:31.206376 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.206258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5b2c47-1a97-4e84-9153-b23a84840b82-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v\" (UID: \"9d5b2c47-1a97-4e84-9153-b23a84840b82\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"
Jul 24 18:18:31.206989 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.206969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5b2c47-1a97-4e84-9153-b23a84840b82-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v\" (UID: \"9d5b2c47-1a97-4e84-9153-b23a84840b82\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"
Jul 24 18:18:31.218253 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.218227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln4l6\" (UniqueName: \"kubernetes.io/projected/9d5b2c47-1a97-4e84-9153-b23a84840b82-kube-api-access-ln4l6\") pod \"payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v\" (UID: \"9d5b2c47-1a97-4e84-9153-b23a84840b82\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"
Jul 24 18:18:31.320161 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.320124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"
Jul 24 18:18:31.485598 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:31.485491    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"]
Jul 24 18:18:31.488119 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:18:31.488093    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5b2c47_1a97_4e84_9153_b23a84840b82.slice/crio-6240733c4a9beaada71d3e8cea12dc2bb2f50c39e9c1dd72fc7e6c64cde9a8f9 WatchSource:0}: Error finding container 6240733c4a9beaada71d3e8cea12dc2bb2f50c39e9c1dd72fc7e6c64cde9a8f9: Status 404 returned error can't find the container with id 6240733c4a9beaada71d3e8cea12dc2bb2f50c39e9c1dd72fc7e6c64cde9a8f9
Jul 24 18:18:32.036092 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:32.035990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v" event={"ID":"9d5b2c47-1a97-4e84-9153-b23a84840b82","Type":"ContainerStarted","Data":"8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4"}
Jul 24 18:18:32.036516 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:32.036128    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"
Jul 24 18:18:32.036516 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:32.036168    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v" event={"ID":"9d5b2c47-1a97-4e84-9153-b23a84840b82","Type":"ContainerStarted","Data":"6240733c4a9beaada71d3e8cea12dc2bb2f50c39e9c1dd72fc7e6c64cde9a8f9"}
Jul 24 18:18:32.037497 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:32.037476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8" event={"ID":"24afbe93-338c-4cb3-ae61-fb780feda33b","Type":"ContainerStarted","Data":"d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7"}
Jul 24 18:18:32.037594 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:32.037538    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:32.054321 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:32.054257    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v" podStartSLOduration=2.054235316 podStartE2EDuration="2.054235316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:18:32.052425651 +0000 UTC m=+2525.568936716" watchObservedRunningTime="2026-07-24 18:18:32.054235316 +0000 UTC m=+2525.570746352"
Jul 24 18:18:32.072800 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:32.072753    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8" podStartSLOduration=1.705722768 podStartE2EDuration="2.072737381s" totalImagesPullingTime="367.014613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:18:32.070939533 +0000 UTC m=+2525.587450571" watchObservedRunningTime="2026-07-24 18:18:32.072737381 +0000 UTC m=+2525.589248416"
Jul 24 18:18:34.464655 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:34.464604    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:34.998608 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:34.998582    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"
Jul 24 18:18:37.013691 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:37.013659    2581 patch_prober.go:28] interesting pod/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 24 18:18:37.014069 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:37.013710    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 18:18:38.049282 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:38.049195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:18:38.071786 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:38.071758    2581 generic.go:356] "Generic (PLEG): container finished" podID="c882666b-2b38-4a49-9f83-9271e7a39681" containerID="5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884" exitCode=0
Jul 24 18:18:38.071937 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:38.071810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" event={"ID":"c882666b-2b38-4a49-9f83-9271e7a39681","Type":"ContainerDied","Data":"5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884"}
Jul 24 18:18:38.072164 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:38.072140    2581 scope.go:117] "RemoveContainer" containerID="5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884"
Jul 24 18:18:38.073345 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:38.073300    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:18:38.073415 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:38.073395    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:18:38.073467 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:38.073436    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:18:39.080784 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:39.080739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" event={"ID":"c882666b-2b38-4a49-9f83-9271e7a39681","Type":"ContainerStarted","Data":"480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb"}
Jul 24 18:18:39.671759 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:39.671729    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:39.672175 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:39.672054    2581 patch_prober.go:28] interesting pod/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 24 18:18:39.672377 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:39.672351    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 24 18:18:40.671720 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:40.671665    2581 patch_prober.go:28] interesting pod/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 24 18:18:40.672349 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:40.672316    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 24 18:18:41.672024 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:41.671983    2581 patch_prober.go:28] interesting pod/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 24 18:18:41.672638 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:41.672600    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 24 18:18:42.671392 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:42.671346    2581 patch_prober.go:28] interesting pod/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 24 18:18:42.671633 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:42.671421    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 24 18:18:43.050924 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:43.050817    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"
Jul 24 18:18:43.674951 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:43.674923    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:43.675288 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:43.675267    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:43.676089 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:43.676064    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:18:45.060770 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:45.060730    2581 patch_prober.go:28] interesting pod/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 24 18:18:45.061151 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:45.060793    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 18:18:46.117986 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:46.117895    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerID="6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb" exitCode=0
Jul 24 18:18:46.118645 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:46.118031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" event={"ID":"cc04ac9a-531b-482b-af64-45306f05eb0a","Type":"ContainerDied","Data":"6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb"}
Jul 24 18:18:46.118645 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:46.118503    2581 scope.go:117] "RemoveContainer" containerID="6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb"
Jul 24 18:18:46.120516 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:46.120487    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:18:46.120773 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:46.120752    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:18:46.120942 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:46.120919    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:18:47.123958 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:47.123923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" event={"ID":"cc04ac9a-531b-482b-af64-45306f05eb0a","Type":"ContainerStarted","Data":"9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986"}
Jul 24 18:18:47.564636 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:47.564592    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:47.564852 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:47.564690    2581 patch_prober.go:28] interesting pod/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 24 18:18:47.564852 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:47.564742    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 24 18:18:48.566126 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:48.566086    2581 patch_prober.go:28] interesting pod/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 24 18:18:48.566544 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:48.566160    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 24 18:18:49.564764 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:49.564723    2581 patch_prober.go:28] interesting pod/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 24 18:18:49.565073 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:49.565042    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 24 18:18:50.564710 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:50.564671    2581 patch_prober.go:28] interesting pod/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 24 18:18:50.565347 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:50.565308    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 24 18:18:51.568051 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:51.568021    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:51.568415 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:51.568368    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:51.569116 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:51.569098    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:18:58.235305 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.235269    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"]
Jul 24 18:18:58.235779 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.235467    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b" podUID="cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421" containerName="maas-api" containerID="cri-o://3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2" gracePeriod=30
Jul 24 18:18:58.272884 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.272858    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:58.294638 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.294609    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"]
Jul 24 18:18:58.295011 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.294831    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b" podUID="e70a3f33-0181-441a-b298-edc10db3872f" containerName="payload-processing" containerID="cri-o://c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3" gracePeriod=30
Jul 24 18:18:58.373748 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.373718    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:18:58.536114 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.536092    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:58.573736 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.573706    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"
Jul 24 18:18:58.647058 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.647033    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-maas-api-tls\") pod \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\" (UID: \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\") "
Jul 24 18:18:58.647210 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.647071    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-openshift-service-ca\") pod \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\" (UID: \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\") "
Jul 24 18:18:58.647210 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.647102    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9kn9\" (UniqueName: \"kubernetes.io/projected/e70a3f33-0181-441a-b298-edc10db3872f-kube-api-access-n9kn9\") pod \"e70a3f33-0181-441a-b298-edc10db3872f\" (UID: \"e70a3f33-0181-441a-b298-edc10db3872f\") "
Jul 24 18:18:58.647210 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.647126    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfpjz\" (UniqueName: \"kubernetes.io/projected/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-kube-api-access-xfpjz\") pod \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\" (UID: \"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421\") "
Jul 24 18:18:58.647210 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.647169    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70a3f33-0181-441a-b298-edc10db3872f-plugins-config-volume\") pod \"e70a3f33-0181-441a-b298-edc10db3872f\" (UID: \"e70a3f33-0181-441a-b298-edc10db3872f\") "
Jul 24 18:18:58.647540 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.647510    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421" (UID: "cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:18:58.647679 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.647648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70a3f33-0181-441a-b298-edc10db3872f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e70a3f33-0181-441a-b298-edc10db3872f" (UID: "e70a3f33-0181-441a-b298-edc10db3872f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:18:58.649067 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.649045    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421" (UID: "cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:18:58.649213 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.649193    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70a3f33-0181-441a-b298-edc10db3872f-kube-api-access-n9kn9" (OuterVolumeSpecName: "kube-api-access-n9kn9") pod "e70a3f33-0181-441a-b298-edc10db3872f" (UID: "e70a3f33-0181-441a-b298-edc10db3872f"). InnerVolumeSpecName "kube-api-access-n9kn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:18:58.649270 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.649222    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-kube-api-access-xfpjz" (OuterVolumeSpecName: "kube-api-access-xfpjz") pod "cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421" (UID: "cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421"). InnerVolumeSpecName "kube-api-access-xfpjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:18:58.748164 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.748073    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70a3f33-0181-441a-b298-edc10db3872f-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:58.748164 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.748104    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:58.748164 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.748117    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:58.748164 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.748128    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9kn9\" (UniqueName: \"kubernetes.io/projected/e70a3f33-0181-441a-b298-edc10db3872f-kube-api-access-n9kn9\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:58.748164 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:58.748140    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfpjz\" (UniqueName: \"kubernetes.io/projected/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421-kube-api-access-xfpjz\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:18:59.176065 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.176028    2581 generic.go:356] "Generic (PLEG): container finished" podID="cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421" containerID="3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2" exitCode=0
Jul 24 18:18:59.176256 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.176085    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"
Jul 24 18:18:59.176256 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.176119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b" event={"ID":"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421","Type":"ContainerDied","Data":"3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2"}
Jul 24 18:18:59.176256 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.176175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b" event={"ID":"cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421","Type":"ContainerDied","Data":"65b4bce308c25c4f991e52edb597f10c4539d5dda523c1f9907e16359b8b2851"}
Jul 24 18:18:59.176256 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.176206    2581 scope.go:117] "RemoveContainer" containerID="3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2"
Jul 24 18:18:59.177427 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.177405    2581 generic.go:356] "Generic (PLEG): container finished" podID="e70a3f33-0181-441a-b298-edc10db3872f" containerID="c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3" exitCode=0
Jul 24 18:18:59.177506 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.177440    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"
Jul 24 18:18:59.177506 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.177456    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b" event={"ID":"e70a3f33-0181-441a-b298-edc10db3872f","Type":"ContainerDied","Data":"c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3"}
Jul 24 18:18:59.177506 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.177493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b" event={"ID":"e70a3f33-0181-441a-b298-edc10db3872f","Type":"ContainerDied","Data":"68fd91737785e9b37a336bbc099809210347fb028fee0568ce3d85ea3185241b"}
Jul 24 18:18:59.185105 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.185089    2581 scope.go:117] "RemoveContainer" containerID="3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2"
Jul 24 18:18:59.185357 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:18:59.185339    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2\": container with ID starting with 3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2 not found: ID does not exist" containerID="3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2"
Jul 24 18:18:59.185416 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.185366    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2"} err="failed to get container status \"3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2\": rpc error: code = NotFound desc = could not find container \"3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2\": container with ID starting with 3c5bf16e2b3be7669c36472c6fb82f1fb98114d03656f3cbf0497ae2c7c1c5b2 not found: ID does not exist"
Jul 24 18:18:59.185416 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.185383    2581 scope.go:117] "RemoveContainer" containerID="c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3"
Jul 24 18:18:59.192479 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.192461    2581 scope.go:117] "RemoveContainer" containerID="c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3"
Jul 24 18:18:59.192733 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:18:59.192715    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3\": container with ID starting with c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3 not found: ID does not exist" containerID="c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3"
Jul 24 18:18:59.192787 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.192741    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3"} err="failed to get container status \"c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3\": rpc error: code = NotFound desc = could not find container \"c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3\": container with ID starting with c386fb31fde6058024629787e948d60a553e677fddcb6619dbc61dd0e6c161d3 not found: ID does not exist"
Jul 24 18:18:59.196611 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.196555    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"]
Jul 24 18:18:59.200709 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.200687    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dcf34d-bc4b5f6fb-82t7b"]
Jul 24 18:18:59.213462 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.213442    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"]
Jul 24 18:18:59.219977 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:18:59.219957    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dcf34d-5fccdc895b-hkw6b"]
Jul 24 18:19:00.958438 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:00.958402    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421" path="/var/lib/kubelet/pods/cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421/volumes"
Jul 24 18:19:00.958831 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:00.958792    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e70a3f33-0181-441a-b298-edc10db3872f" path="/var/lib/kubelet/pods/e70a3f33-0181-441a-b298-edc10db3872f/volumes"
Jul 24 18:19:02.814478 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:02.814444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:03.195889 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:03.195803    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:09.198146 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:09.198113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:09.266765 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:09.266734    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:09.430548 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:09.430515    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"]
Jul 24 18:19:09.430847 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:09.430747    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy" containerID="cri-o://480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb" gracePeriod=30
Jul 24 18:19:09.453065 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:09.453001    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:09.702771 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:09.702733    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:13.677343 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:13.677298    2581 patch_prober.go:28] interesting pod/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 18:19:13.677720 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:13.677375    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 18:19:14.678221 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.678198    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:19:14.783798 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.783720    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c882666b-2b38-4a49-9f83-9271e7a39681-istiod-ca-cert\") pod \"c882666b-2b38-4a49-9f83-9271e7a39681\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") "
Jul 24 18:19:14.783798 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.783777    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c882666b-2b38-4a49-9f83-9271e7a39681-istio-token\") pod \"c882666b-2b38-4a49-9f83-9271e7a39681\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") "
Jul 24 18:19:14.784013 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.783807    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-workload-socket\") pod \"c882666b-2b38-4a49-9f83-9271e7a39681\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") "
Jul 24 18:19:14.784013 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.783844    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c882666b-2b38-4a49-9f83-9271e7a39681-istio-podinfo\") pod \"c882666b-2b38-4a49-9f83-9271e7a39681\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") "
Jul 24 18:19:14.784013 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.783893    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-istio-envoy\") pod \"c882666b-2b38-4a49-9f83-9271e7a39681\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") "
Jul 24 18:19:14.784013 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.783916    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-istio-data\") pod \"c882666b-2b38-4a49-9f83-9271e7a39681\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") "
Jul 24 18:19:14.784013 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.783951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-workload-certs\") pod \"c882666b-2b38-4a49-9f83-9271e7a39681\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") "
Jul 24 18:19:14.784013 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.783984    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-credential-socket\") pod \"c882666b-2b38-4a49-9f83-9271e7a39681\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") "
Jul 24 18:19:14.784323 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.784054    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjxvf\" (UniqueName: \"kubernetes.io/projected/c882666b-2b38-4a49-9f83-9271e7a39681-kube-api-access-hjxvf\") pod \"c882666b-2b38-4a49-9f83-9271e7a39681\" (UID: \"c882666b-2b38-4a49-9f83-9271e7a39681\") "
Jul 24 18:19:14.784323 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.784138    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c882666b-2b38-4a49-9f83-9271e7a39681-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c882666b-2b38-4a49-9f83-9271e7a39681" (UID: "c882666b-2b38-4a49-9f83-9271e7a39681"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:14.784608 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.784433    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c882666b-2b38-4a49-9f83-9271e7a39681-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.784608 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.784511    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c882666b-2b38-4a49-9f83-9271e7a39681" (UID: "c882666b-2b38-4a49-9f83-9271e7a39681"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:14.784861 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.784831    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c882666b-2b38-4a49-9f83-9271e7a39681" (UID: "c882666b-2b38-4a49-9f83-9271e7a39681"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:14.785072 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.785046    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c882666b-2b38-4a49-9f83-9271e7a39681" (UID: "c882666b-2b38-4a49-9f83-9271e7a39681"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:14.785072 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.785059    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-istio-data" (OuterVolumeSpecName: "istio-data") pod "c882666b-2b38-4a49-9f83-9271e7a39681" (UID: "c882666b-2b38-4a49-9f83-9271e7a39681"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:14.786329 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.786304    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c882666b-2b38-4a49-9f83-9271e7a39681-kube-api-access-hjxvf" (OuterVolumeSpecName: "kube-api-access-hjxvf") pod "c882666b-2b38-4a49-9f83-9271e7a39681" (UID: "c882666b-2b38-4a49-9f83-9271e7a39681"). InnerVolumeSpecName "kube-api-access-hjxvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:14.786441 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.786417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c882666b-2b38-4a49-9f83-9271e7a39681" (UID: "c882666b-2b38-4a49-9f83-9271e7a39681"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:14.786505 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.786425    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c882666b-2b38-4a49-9f83-9271e7a39681-istio-token" (OuterVolumeSpecName: "istio-token") pod "c882666b-2b38-4a49-9f83-9271e7a39681" (UID: "c882666b-2b38-4a49-9f83-9271e7a39681"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:14.786505 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.786498    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c882666b-2b38-4a49-9f83-9271e7a39681-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c882666b-2b38-4a49-9f83-9271e7a39681" (UID: "c882666b-2b38-4a49-9f83-9271e7a39681"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 18:19:14.885732 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.885708    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c882666b-2b38-4a49-9f83-9271e7a39681-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.885732 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.885729    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.885732 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.885737    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c882666b-2b38-4a49-9f83-9271e7a39681-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.885915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.885746    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.885915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.885755    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.885915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.885762    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.885915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.885770    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c882666b-2b38-4a49-9f83-9271e7a39681-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.885915 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:14.885778    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjxvf\" (UniqueName: \"kubernetes.io/projected/c882666b-2b38-4a49-9f83-9271e7a39681-kube-api-access-hjxvf\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:15.251748 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:15.251714    2581 generic.go:356] "Generic (PLEG): container finished" podID="c882666b-2b38-4a49-9f83-9271e7a39681" containerID="480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb" exitCode=0
Jul 24 18:19:15.251932 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:15.251766    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"
Jul 24 18:19:15.251932 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:15.251796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" event={"ID":"c882666b-2b38-4a49-9f83-9271e7a39681","Type":"ContainerDied","Data":"480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb"}
Jul 24 18:19:15.251932 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:15.251847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl" event={"ID":"c882666b-2b38-4a49-9f83-9271e7a39681","Type":"ContainerDied","Data":"f5da9de9e7c1d715a4ed0d8fdbe0a1d5c121167c6cf0234a6c1bb3de60859fbb"}
Jul 24 18:19:15.251932 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:15.251870    2581 scope.go:117] "RemoveContainer" containerID="480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb"
Jul 24 18:19:15.260221 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:15.260203    2581 scope.go:117] "RemoveContainer" containerID="5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884"
Jul 24 18:19:15.270612 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:15.270587    2581 scope.go:117] "RemoveContainer" containerID="480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb"
Jul 24 18:19:15.271128 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:19:15.271082    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb\": container with ID starting with 480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb not found: ID does not exist" containerID="480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb"
Jul 24 18:19:15.271329 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:15.271301    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb"} err="failed to get container status \"480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb\": rpc error: code = NotFound desc = could not find container \"480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb\": container with ID starting with 480b1ca4f4e7b60393032e1736e15e3f0549663b89e6bf7b8216757be3a7f2fb not found: ID does not exist"
Jul 24 18:19:15.271456 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:15.271421    2581 scope.go:117] "RemoveContainer" containerID="5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884"
Jul 24 18:19:15.271746 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:19:15.271722    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884\": container with ID starting with 5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884 not found: ID does not exist" containerID="5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884"
Jul 24 18:19:15.271828 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:15.271754    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884"} err="failed to get container status \"5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884\": rpc error: code = NotFound desc = could not find container \"5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884\": container with ID starting with 5a8e6f9f7ceaafad3dd0f41c2376bfe14352b1db9b7cb2a62ebb3f5fa0a64884 not found: ID does not exist"
Jul 24 18:19:15.272894 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:15.272875    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"]
Jul 24 18:19:15.282409 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:15.282385    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dcf34d-openshift-default-6bb79cc7d4-97pxl"]
Jul 24 18:19:16.958116 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:16.958083    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" path="/var/lib/kubelet/pods/c882666b-2b38-4a49-9f83-9271e7a39681/volumes"
Jul 24 18:19:25.292250 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.292211    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"]
Jul 24 18:19:25.292707 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.292528    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8" podUID="24afbe93-338c-4cb3-ae61-fb780feda33b" containerName="maas-api" containerID="cri-o://d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7" gracePeriod=30
Jul 24 18:19:25.323038 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.323008    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:25.370968 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.370925    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"]
Jul 24 18:19:25.371228 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.371197    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v" podUID="9d5b2c47-1a97-4e84-9153-b23a84840b82" containerName="payload-pre-processing" containerID="cri-o://8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4" gracePeriod=30
Jul 24 18:19:25.421172 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.421140    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:25.524332 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.524297    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:25.614194 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.614172    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:19:25.646728 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.646703    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"
Jul 24 18:19:25.772593 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.772538    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln4l6\" (UniqueName: \"kubernetes.io/projected/9d5b2c47-1a97-4e84-9153-b23a84840b82-kube-api-access-ln4l6\") pod \"9d5b2c47-1a97-4e84-9153-b23a84840b82\" (UID: \"9d5b2c47-1a97-4e84-9153-b23a84840b82\") "
Jul 24 18:19:25.772785 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.772601    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gczg6\" (UniqueName: \"kubernetes.io/projected/24afbe93-338c-4cb3-ae61-fb780feda33b-kube-api-access-gczg6\") pod \"24afbe93-338c-4cb3-ae61-fb780feda33b\" (UID: \"24afbe93-338c-4cb3-ae61-fb780feda33b\") "
Jul 24 18:19:25.772785 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.772712    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24afbe93-338c-4cb3-ae61-fb780feda33b-maas-api-tls\") pod \"24afbe93-338c-4cb3-ae61-fb780feda33b\" (UID: \"24afbe93-338c-4cb3-ae61-fb780feda33b\") "
Jul 24 18:19:25.772919 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.772805    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24afbe93-338c-4cb3-ae61-fb780feda33b-openshift-service-ca\") pod \"24afbe93-338c-4cb3-ae61-fb780feda33b\" (UID: \"24afbe93-338c-4cb3-ae61-fb780feda33b\") "
Jul 24 18:19:25.772919 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.772828    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5b2c47-1a97-4e84-9153-b23a84840b82-plugins-config-volume\") pod \"9d5b2c47-1a97-4e84-9153-b23a84840b82\" (UID: \"9d5b2c47-1a97-4e84-9153-b23a84840b82\") "
Jul 24 18:19:25.773203 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.773169    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24afbe93-338c-4cb3-ae61-fb780feda33b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24afbe93-338c-4cb3-ae61-fb780feda33b" (UID: "24afbe93-338c-4cb3-ae61-fb780feda33b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:25.773317 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.773230    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5b2c47-1a97-4e84-9153-b23a84840b82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d5b2c47-1a97-4e84-9153-b23a84840b82" (UID: "9d5b2c47-1a97-4e84-9153-b23a84840b82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:25.774737 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.774695    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24afbe93-338c-4cb3-ae61-fb780feda33b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24afbe93-338c-4cb3-ae61-fb780feda33b" (UID: "24afbe93-338c-4cb3-ae61-fb780feda33b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:19:25.774852 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.774794    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24afbe93-338c-4cb3-ae61-fb780feda33b-kube-api-access-gczg6" (OuterVolumeSpecName: "kube-api-access-gczg6") pod "24afbe93-338c-4cb3-ae61-fb780feda33b" (UID: "24afbe93-338c-4cb3-ae61-fb780feda33b"). InnerVolumeSpecName "kube-api-access-gczg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:25.774852 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.774823    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5b2c47-1a97-4e84-9153-b23a84840b82-kube-api-access-ln4l6" (OuterVolumeSpecName: "kube-api-access-ln4l6") pod "9d5b2c47-1a97-4e84-9153-b23a84840b82" (UID: "9d5b2c47-1a97-4e84-9153-b23a84840b82"). InnerVolumeSpecName "kube-api-access-ln4l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:25.874251 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.874186    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24afbe93-338c-4cb3-ae61-fb780feda33b-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:25.874251 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.874210    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5b2c47-1a97-4e84-9153-b23a84840b82-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:25.874251 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.874220    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln4l6\" (UniqueName: \"kubernetes.io/projected/9d5b2c47-1a97-4e84-9153-b23a84840b82-kube-api-access-ln4l6\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:25.874251 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.874228    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gczg6\" (UniqueName: \"kubernetes.io/projected/24afbe93-338c-4cb3-ae61-fb780feda33b-kube-api-access-gczg6\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:25.874251 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:25.874238    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24afbe93-338c-4cb3-ae61-fb780feda33b-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:26.301262 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.301170    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d5b2c47-1a97-4e84-9153-b23a84840b82" containerID="8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4" exitCode=0
Jul 24 18:19:26.301262 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.301228    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"
Jul 24 18:19:26.301772 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.301256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v" event={"ID":"9d5b2c47-1a97-4e84-9153-b23a84840b82","Type":"ContainerDied","Data":"8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4"}
Jul 24 18:19:26.301772 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.301305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v" event={"ID":"9d5b2c47-1a97-4e84-9153-b23a84840b82","Type":"ContainerDied","Data":"6240733c4a9beaada71d3e8cea12dc2bb2f50c39e9c1dd72fc7e6c64cde9a8f9"}
Jul 24 18:19:26.301772 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.301336    2581 scope.go:117] "RemoveContainer" containerID="8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4"
Jul 24 18:19:26.302701 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.302675    2581 generic.go:356] "Generic (PLEG): container finished" podID="24afbe93-338c-4cb3-ae61-fb780feda33b" containerID="d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7" exitCode=0
Jul 24 18:19:26.302819 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.302705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8" event={"ID":"24afbe93-338c-4cb3-ae61-fb780feda33b","Type":"ContainerDied","Data":"d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7"}
Jul 24 18:19:26.302819 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.302721    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"
Jul 24 18:19:26.302819 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.302768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8" event={"ID":"24afbe93-338c-4cb3-ae61-fb780feda33b","Type":"ContainerDied","Data":"3aaeb02df1ed2307a1ce748f3cb8b28f99a3a035a837d3a3b629e93bc31de0b1"}
Jul 24 18:19:26.314192 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.314180    2581 scope.go:117] "RemoveContainer" containerID="8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4"
Jul 24 18:19:26.314424 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:19:26.314407    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4\": container with ID starting with 8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4 not found: ID does not exist" containerID="8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4"
Jul 24 18:19:26.314484 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.314430    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4"} err="failed to get container status \"8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4\": rpc error: code = NotFound desc = could not find container \"8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4\": container with ID starting with 8a697559b592f0421505f0294978b021022045745d4e013d2932d4747bfafae4 not found: ID does not exist"
Jul 24 18:19:26.314484 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.314445    2581 scope.go:117] "RemoveContainer" containerID="d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7"
Jul 24 18:19:26.321445 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.321428    2581 scope.go:117] "RemoveContainer" containerID="d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7"
Jul 24 18:19:26.321703 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:19:26.321684    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7\": container with ID starting with d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7 not found: ID does not exist" containerID="d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7"
Jul 24 18:19:26.321751 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.321710    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7"} err="failed to get container status \"d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7\": rpc error: code = NotFound desc = could not find container \"d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7\": container with ID starting with d3b340dc5103ea218c7865dc8fe57356638cc66327333e130d4b28722a2a41d7 not found: ID does not exist"
Jul 24 18:19:26.327675 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.327657    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"]
Jul 24 18:19:26.329101 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.329082    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4a14d5-7dcc99bcb4-rfgc8"]
Jul 24 18:19:26.339069 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.339049    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"]
Jul 24 18:19:26.342796 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.342775    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4a14d5-5bd4b4bbc6-x496v"]
Jul 24 18:19:26.958426 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.958397    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24afbe93-338c-4cb3-ae61-fb780feda33b" path="/var/lib/kubelet/pods/24afbe93-338c-4cb3-ae61-fb780feda33b/volumes"
Jul 24 18:19:26.958763 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:26.958750    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d5b2c47-1a97-4e84-9153-b23a84840b82" path="/var/lib/kubelet/pods/9d5b2c47-1a97-4e84-9153-b23a84840b82/volumes"
Jul 24 18:19:29.895232 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:29.895189    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:30.285240 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:30.285151    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:36.030028 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:36.029992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:36.125877 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:36.125842    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:36.290447 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:36.290354    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"]
Jul 24 18:19:36.290650 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:36.290622    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy" containerID="cri-o://9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986" gracePeriod=30
Jul 24 18:19:36.299771 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:36.299735    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:36.530165 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:36.530136    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:36.571260 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:36.571226    2581 patch_prober.go:28] interesting pod/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 18:19:36.571397 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:36.571298    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 18:19:36.888037 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:36.887955    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:37.204077 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204006    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"]
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204276    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy"
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204286    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy"
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204298    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy"
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204304    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy"
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204313    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421" containerName="maas-api"
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204318    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421" containerName="maas-api"
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204325    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24afbe93-338c-4cb3-ae61-fb780feda33b" containerName="maas-api"
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204330    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="24afbe93-338c-4cb3-ae61-fb780feda33b" containerName="maas-api"
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204339    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70a3f33-0181-441a-b298-edc10db3872f" containerName="payload-processing"
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204344    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70a3f33-0181-441a-b298-edc10db3872f" containerName="payload-processing"
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204350    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d5b2c47-1a97-4e84-9153-b23a84840b82" containerName="payload-pre-processing"
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204355    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d5b2c47-1a97-4e84-9153-b23a84840b82" containerName="payload-pre-processing"
Jul 24 18:19:37.204402 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204404    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy"
Jul 24 18:19:37.204970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204413    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d5b2c47-1a97-4e84-9153-b23a84840b82" containerName="payload-pre-processing"
Jul 24 18:19:37.204970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204420    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c882666b-2b38-4a49-9f83-9271e7a39681" containerName="istio-proxy"
Jul 24 18:19:37.204970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204425    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd9f71d9-6ca3-4c5e-8ddb-1eec243a8421" containerName="maas-api"
Jul 24 18:19:37.204970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204430    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="24afbe93-338c-4cb3-ae61-fb780feda33b" containerName="maas-api"
Jul 24 18:19:37.204970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.204435    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70a3f33-0181-441a-b298-edc10db3872f" containerName="payload-processing"
Jul 24 18:19:37.207247 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.207230    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.213723 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.213389    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-100cff-openshift-default-dockercfg-lsd2n\""
Jul 24 18:19:37.251162 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.251139    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"]
Jul 24 18:19:37.255239 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.255217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-workload-certs\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.255357 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.255251    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-token\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.255357 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.255279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-credential-socket\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.255448 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.255353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rwcs\" (UniqueName: \"kubernetes.io/projected/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-kube-api-access-2rwcs\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.255448 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.255390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-workload-socket\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.255448 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.255418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istiod-ca-cert\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.255448 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.255441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-podinfo\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.255645 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.255460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-envoy\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.255645 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.255559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-data\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.264667 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.264645    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:37.356648 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.356614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-data\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.356648 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.356653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-workload-certs\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.356869 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.356675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-token\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.356869 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.356699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-credential-socket\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.356869 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.356743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rwcs\" (UniqueName: \"kubernetes.io/projected/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-kube-api-access-2rwcs\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.356869 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.356773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-workload-socket\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.356869 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.356800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istiod-ca-cert\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.356869 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.356822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-podinfo\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.357255 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.356984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-envoy\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.357255 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.357069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-data\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.357255 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.357104    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-workload-certs\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.357255 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.357136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-workload-socket\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.357400 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.357335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-credential-socket\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.357525 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.357507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istiod-ca-cert\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.359116 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.359096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-envoy\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.359341 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.359323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-podinfo\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.381005 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.380970    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rwcs\" (UniqueName: \"kubernetes.io/projected/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-kube-api-access-2rwcs\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.383301 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.383281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-token\") pod \"e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") " pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.517364 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.517272    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:37.649525 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.649311    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"]
Jul 24 18:19:37.651593 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:19:37.651550    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233b69e0_5dd4_4235_a7f4_3bff6c32f89c.slice/crio-bf8b87f006a3803505d90e0dfed69ab3094e0f526e03fd45c4d9a5ff1e87d20e WatchSource:0}: Error finding container bf8b87f006a3803505d90e0dfed69ab3094e0f526e03fd45c4d9a5ff1e87d20e: Status 404 returned error can't find the container with id bf8b87f006a3803505d90e0dfed69ab3094e0f526e03fd45c4d9a5ff1e87d20e
Jul 24 18:19:37.651851 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.651832    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:37.653688 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.653660    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:19:37.653774 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.653735    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:19:37.653832 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:37.653779    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:19:38.353865 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:38.353827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb" event={"ID":"233b69e0-5dd4-4235-a7f4-3bff6c32f89c","Type":"ContainerStarted","Data":"942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5"}
Jul 24 18:19:38.354233 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:38.353885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb" event={"ID":"233b69e0-5dd4-4235-a7f4-3bff6c32f89c","Type":"ContainerStarted","Data":"bf8b87f006a3803505d90e0dfed69ab3094e0f526e03fd45c4d9a5ff1e87d20e"}
Jul 24 18:19:38.375165 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:38.375119    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb" podStartSLOduration=1.375103974 podStartE2EDuration="1.375103974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:19:38.37336865 +0000 UTC m=+2591.889879686" watchObservedRunningTime="2026-07-24 18:19:38.375103974 +0000 UTC m=+2591.891615010"
Jul 24 18:19:38.517798 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:38.517769    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:38.522535 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:38.522513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:39.363522 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:39.363488    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:39.364507 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:39.364488    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:40.326125 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.326079    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:40.380391 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.380363    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"]
Jul 24 18:19:40.383465 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.383443    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:40.385919 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.385894    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:19:40.386082 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.386059    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-100cff-serving-cert\""
Jul 24 18:19:40.387156 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.387141    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fvm85\""
Jul 24 18:19:40.387523 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.387499    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:19:40.399022 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.398999    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"]
Jul 24 18:19:40.415772 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.415743    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:40.485112 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.485077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86cebe-90d2-4940-b323-eddcc6c06c5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:40.485331 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.485300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b86cebe-90d2-4940-b323-eddcc6c06c5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:40.485421 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.485376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw4dn\" (UniqueName: \"kubernetes.io/projected/7b86cebe-90d2-4940-b323-eddcc6c06c5a-kube-api-access-qw4dn\") pod \"maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:40.497607 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.497539    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:40.586463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.586366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86cebe-90d2-4940-b323-eddcc6c06c5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:40.586463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.586442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b86cebe-90d2-4940-b323-eddcc6c06c5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:40.586722 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.586467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw4dn\" (UniqueName: \"kubernetes.io/projected/7b86cebe-90d2-4940-b323-eddcc6c06c5a-kube-api-access-qw4dn\") pod \"maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:40.586722 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:19:40.586546    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-serving-cert: secret "maas-api-e2e-shared-a-100cff-serving-cert" not found
Jul 24 18:19:40.586722 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:19:40.586653    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b86cebe-90d2-4940-b323-eddcc6c06c5a-maas-api-tls podName:7b86cebe-90d2-4940-b323-eddcc6c06c5a nodeName:}" failed. No retries permitted until 2026-07-24 18:19:41.086631672 +0000 UTC m=+2594.603142686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b86cebe-90d2-4940-b323-eddcc6c06c5a-maas-api-tls") pod "maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6" (UID: "7b86cebe-90d2-4940-b323-eddcc6c06c5a") : secret "maas-api-e2e-shared-a-100cff-serving-cert" not found
Jul 24 18:19:40.587088 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.587072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86cebe-90d2-4940-b323-eddcc6c06c5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:40.595366 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.595347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw4dn\" (UniqueName: \"kubernetes.io/projected/7b86cebe-90d2-4940-b323-eddcc6c06c5a-kube-api-access-qw4dn\") pod \"maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:40.789536 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.789500    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:40.935933 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:40.935851    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:41.033199 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.033162    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:41.091848 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.091813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b86cebe-90d2-4940-b323-eddcc6c06c5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:41.094169 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.094139    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b86cebe-90d2-4940-b323-eddcc6c06c5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:41.107507 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.107483    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"]
Jul 24 18:19:41.110902 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.110887    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"
Jul 24 18:19:41.114352 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.114331    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-100cff-dockercfg-jgkd2\""
Jul 24 18:19:41.114469 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.114458    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-100cff\""
Jul 24 18:19:41.122372 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.122353    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"]
Jul 24 18:19:41.192463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.192389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkkzv\" (UniqueName: \"kubernetes.io/projected/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f-kube-api-access-xkkzv\") pod \"payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb\" (UID: \"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"
Jul 24 18:19:41.192463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.192440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb\" (UID: \"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"
Jul 24 18:19:41.204257 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.204227    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"]
Jul 24 18:19:41.207441 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.207421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"
Jul 24 18:19:41.217928 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.217907    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"]
Jul 24 18:19:41.293719 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.293686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb\" (UID: \"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"
Jul 24 18:19:41.293877 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.293752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:41.293877 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.293773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crbx5\" (UniqueName: \"kubernetes.io/projected/e24f0986-e94e-4538-b1ff-d9df35e8445f-kube-api-access-crbx5\") pod \"payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw\" (UID: \"e24f0986-e94e-4538-b1ff-d9df35e8445f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"
Jul 24 18:19:41.293877 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.293813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24f0986-e94e-4538-b1ff-d9df35e8445f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw\" (UID: \"e24f0986-e94e-4538-b1ff-d9df35e8445f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"
Jul 24 18:19:41.293877 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.293847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkkzv\" (UniqueName: \"kubernetes.io/projected/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f-kube-api-access-xkkzv\") pod \"payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb\" (UID: \"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"
Jul 24 18:19:41.294397 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.294380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb\" (UID: \"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"
Jul 24 18:19:41.302861 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.302835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkkzv\" (UniqueName: \"kubernetes.io/projected/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f-kube-api-access-xkkzv\") pod \"payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb\" (UID: \"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"
Jul 24 18:19:41.394473 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.394438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24f0986-e94e-4538-b1ff-d9df35e8445f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw\" (UID: \"e24f0986-e94e-4538-b1ff-d9df35e8445f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"
Jul 24 18:19:41.394929 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.394664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crbx5\" (UniqueName: \"kubernetes.io/projected/e24f0986-e94e-4538-b1ff-d9df35e8445f-kube-api-access-crbx5\") pod \"payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw\" (UID: \"e24f0986-e94e-4538-b1ff-d9df35e8445f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"
Jul 24 18:19:41.395219 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.395197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24f0986-e94e-4538-b1ff-d9df35e8445f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw\" (UID: \"e24f0986-e94e-4538-b1ff-d9df35e8445f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"
Jul 24 18:19:41.404244 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.404216    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crbx5\" (UniqueName: \"kubernetes.io/projected/e24f0986-e94e-4538-b1ff-d9df35e8445f-kube-api-access-crbx5\") pod \"payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw\" (UID: \"e24f0986-e94e-4538-b1ff-d9df35e8445f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"
Jul 24 18:19:41.421041 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.421017    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"
Jul 24 18:19:41.430245 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.430221    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"]
Jul 24 18:19:41.431745 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:19:41.431639    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b86cebe_90d2_4940_b323_eddcc6c06c5a.slice/crio-e3d16e3960e91d516e9ba15b50b9a1cdba4d211307201c43700e1a9da5b44083 WatchSource:0}: Error finding container e3d16e3960e91d516e9ba15b50b9a1cdba4d211307201c43700e1a9da5b44083: Status 404 returned error can't find the container with id e3d16e3960e91d516e9ba15b50b9a1cdba4d211307201c43700e1a9da5b44083
Jul 24 18:19:41.518994 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.518965    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"
Jul 24 18:19:41.542069 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.542042    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:19:41.560546 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.560514    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"]
Jul 24 18:19:41.565943 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:19:41.565859    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ba7042_c1f7_44cf_9604_dbb2ae0ae94f.slice/crio-fc86fa781c407d1508fc36845bc75bfb46a57040d0df810cfab6be60621c5e38 WatchSource:0}: Error finding container fc86fa781c407d1508fc36845bc75bfb46a57040d0df810cfab6be60621c5e38: Status 404 returned error can't find the container with id fc86fa781c407d1508fc36845bc75bfb46a57040d0df810cfab6be60621c5e38
Jul 24 18:19:41.596136 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.596106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-envoy\") pod \"cc04ac9a-531b-482b-af64-45306f05eb0a\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") "
Jul 24 18:19:41.596264 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.596144    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn2cj\" (UniqueName: \"kubernetes.io/projected/cc04ac9a-531b-482b-af64-45306f05eb0a-kube-api-access-pn2cj\") pod \"cc04ac9a-531b-482b-af64-45306f05eb0a\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") "
Jul 24 18:19:41.596264 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.596171    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-workload-socket\") pod \"cc04ac9a-531b-482b-af64-45306f05eb0a\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") "
Jul 24 18:19:41.596264 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.596202    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-podinfo\") pod \"cc04ac9a-531b-482b-af64-45306f05eb0a\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") "
Jul 24 18:19:41.596264 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.596258    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-workload-certs\") pod \"cc04ac9a-531b-482b-af64-45306f05eb0a\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") "
Jul 24 18:19:41.596404 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.596297    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-credential-socket\") pod \"cc04ac9a-531b-482b-af64-45306f05eb0a\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") "
Jul 24 18:19:41.596404 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.596325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-data\") pod \"cc04ac9a-531b-482b-af64-45306f05eb0a\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") "
Jul 24 18:19:41.596404 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.596369    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-token\") pod \"cc04ac9a-531b-482b-af64-45306f05eb0a\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") "
Jul 24 18:19:41.596404 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.596394    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc04ac9a-531b-482b-af64-45306f05eb0a-istiod-ca-cert\") pod \"cc04ac9a-531b-482b-af64-45306f05eb0a\" (UID: \"cc04ac9a-531b-482b-af64-45306f05eb0a\") "
Jul 24 18:19:41.596822 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.596614    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc04ac9a-531b-482b-af64-45306f05eb0a" (UID: "cc04ac9a-531b-482b-af64-45306f05eb0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:41.596822 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.596774    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc04ac9a-531b-482b-af64-45306f05eb0a" (UID: "cc04ac9a-531b-482b-af64-45306f05eb0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:41.597294 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.597212    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc04ac9a-531b-482b-af64-45306f05eb0a" (UID: "cc04ac9a-531b-482b-af64-45306f05eb0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:41.597705 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.597553    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc04ac9a-531b-482b-af64-45306f05eb0a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc04ac9a-531b-482b-af64-45306f05eb0a" (UID: "cc04ac9a-531b-482b-af64-45306f05eb0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:41.597856 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.597829    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc04ac9a-531b-482b-af64-45306f05eb0a" (UID: "cc04ac9a-531b-482b-af64-45306f05eb0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:41.598597 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.598553    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc04ac9a-531b-482b-af64-45306f05eb0a" (UID: "cc04ac9a-531b-482b-af64-45306f05eb0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:41.598876 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.598855    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc04ac9a-531b-482b-af64-45306f05eb0a-kube-api-access-pn2cj" (OuterVolumeSpecName: "kube-api-access-pn2cj") pod "cc04ac9a-531b-482b-af64-45306f05eb0a" (UID: "cc04ac9a-531b-482b-af64-45306f05eb0a"). InnerVolumeSpecName "kube-api-access-pn2cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:41.599493 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.599471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc04ac9a-531b-482b-af64-45306f05eb0a" (UID: "cc04ac9a-531b-482b-af64-45306f05eb0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 18:19:41.599970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.599948    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc04ac9a-531b-482b-af64-45306f05eb0a" (UID: "cc04ac9a-531b-482b-af64-45306f05eb0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:41.665219 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.665017    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"]
Jul 24 18:19:41.667381 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:19:41.667357    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24f0986_e94e_4538_b1ff_d9df35e8445f.slice/crio-d076caaa1048b2e84ecc7f7ae3275854ab1100793feed13cfb3e04c28d6e285d WatchSource:0}: Error finding container d076caaa1048b2e84ecc7f7ae3275854ab1100793feed13cfb3e04c28d6e285d: Status 404 returned error can't find the container with id d076caaa1048b2e84ecc7f7ae3275854ab1100793feed13cfb3e04c28d6e285d
Jul 24 18:19:41.697267 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.697245    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:41.697267 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.697270    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn2cj\" (UniqueName: \"kubernetes.io/projected/cc04ac9a-531b-482b-af64-45306f05eb0a-kube-api-access-pn2cj\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:41.697433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.697284    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:41.697433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.697300    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:41.697433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.697314    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:41.697433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.697332    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:41.697433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.697344    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:41.697433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.697357    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc04ac9a-531b-482b-af64-45306f05eb0a-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:41.697433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:41.697370    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc04ac9a-531b-482b-af64-45306f05eb0a-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:19:42.378109 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.378073    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerID="9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986" exitCode=0
Jul 24 18:19:42.378109 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.378117    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"
Jul 24 18:19:42.378311 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.378158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" event={"ID":"cc04ac9a-531b-482b-af64-45306f05eb0a","Type":"ContainerDied","Data":"9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986"}
Jul 24 18:19:42.378311 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.378209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj" event={"ID":"cc04ac9a-531b-482b-af64-45306f05eb0a","Type":"ContainerDied","Data":"e8294d741f1fbd5f919edcfb1772b27292bd2f1f0eeaa9aafa131df527314d09"}
Jul 24 18:19:42.378311 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.378232    2581 scope.go:117] "RemoveContainer" containerID="9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986"
Jul 24 18:19:42.379902 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.379879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw" event={"ID":"e24f0986-e94e-4538-b1ff-d9df35e8445f","Type":"ContainerStarted","Data":"991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278"}
Jul 24 18:19:42.380019 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.379926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw" event={"ID":"e24f0986-e94e-4538-b1ff-d9df35e8445f","Type":"ContainerStarted","Data":"d076caaa1048b2e84ecc7f7ae3275854ab1100793feed13cfb3e04c28d6e285d"}
Jul 24 18:19:42.380019 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.380006    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"
Jul 24 18:19:42.381656 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.381631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6" event={"ID":"7b86cebe-90d2-4940-b323-eddcc6c06c5a","Type":"ContainerStarted","Data":"0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25"}
Jul 24 18:19:42.381786 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.381683    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:42.381786 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.381707    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6" event={"ID":"7b86cebe-90d2-4940-b323-eddcc6c06c5a","Type":"ContainerStarted","Data":"e3d16e3960e91d516e9ba15b50b9a1cdba4d211307201c43700e1a9da5b44083"}
Jul 24 18:19:42.383215 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.383191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb" event={"ID":"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f","Type":"ContainerStarted","Data":"f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5"}
Jul 24 18:19:42.383323 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.383235    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb" event={"ID":"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f","Type":"ContainerStarted","Data":"fc86fa781c407d1508fc36845bc75bfb46a57040d0df810cfab6be60621c5e38"}
Jul 24 18:19:42.383323 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.383266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"
Jul 24 18:19:42.388137 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.388122    2581 scope.go:117] "RemoveContainer" containerID="6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb"
Jul 24 18:19:42.395593 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.395578    2581 scope.go:117] "RemoveContainer" containerID="9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986"
Jul 24 18:19:42.395831 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:19:42.395800    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986\": container with ID starting with 9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986 not found: ID does not exist" containerID="9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986"
Jul 24 18:19:42.395831 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.395823    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986"} err="failed to get container status \"9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986\": rpc error: code = NotFound desc = could not find container \"9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986\": container with ID starting with 9a1da81db56a8ac727ba7d1e6b43cf7cd036ae8f41005e7339c6e80951fc3986 not found: ID does not exist"
Jul 24 18:19:42.395907 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.395838    2581 scope.go:117] "RemoveContainer" containerID="6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb"
Jul 24 18:19:42.396056 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:19:42.396037    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb\": container with ID starting with 6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb not found: ID does not exist" containerID="6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb"
Jul 24 18:19:42.396094 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.396062    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb"} err="failed to get container status \"6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb\": rpc error: code = NotFound desc = could not find container \"6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb\": container with ID starting with 6b92a086ce56fbff6481b2745a3ff2aeb2e42bbc68b4b3455e398d1a33cf16eb not found: ID does not exist"
Jul 24 18:19:42.423082 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.423036    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6" podStartSLOduration=2.04725728 podStartE2EDuration="2.423023269s" totalImagesPullingTime="375.765989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:19:42.402900042 +0000 UTC m=+2595.919411077" watchObservedRunningTime="2026-07-24 18:19:42.423023269 +0000 UTC m=+2595.939534306"
Jul 24 18:19:42.423186 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.423126    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw" podStartSLOduration=1.423120413 podStartE2EDuration="1.423120413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:19:42.42154469 +0000 UTC m=+2595.938055745" watchObservedRunningTime="2026-07-24 18:19:42.423120413 +0000 UTC m=+2595.939631427"
Jul 24 18:19:42.437588 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.437527    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb" podStartSLOduration=1.437513533 podStartE2EDuration="1.437513533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:19:42.435723669 +0000 UTC m=+2595.952234709" watchObservedRunningTime="2026-07-24 18:19:42.437513533 +0000 UTC m=+2595.954024568"
Jul 24 18:19:42.452936 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.452905    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"]
Jul 24 18:19:42.457366 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.457338    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4a14d5-openshift-default-5d7cf6647b-wzkdj"]
Jul 24 18:19:42.958318 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:42.958287    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" path="/var/lib/kubelet/pods/cc04ac9a-531b-482b-af64-45306f05eb0a/volumes"
Jul 24 18:19:44.837409 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:44.837369    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:45.184709 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.184621    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:45.476346 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.476265    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"]
Jul 24 18:19:45.476637 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.476623    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy"
Jul 24 18:19:45.476693 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.476639    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy"
Jul 24 18:19:45.476693 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.476647    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy"
Jul 24 18:19:45.476693 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.476652    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy"
Jul 24 18:19:45.476797 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.476707    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy"
Jul 24 18:19:45.481082 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.481064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.483537 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.483518    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-c4d732-openshift-default-dockercfg-v924h\""
Jul 24 18:19:45.492080 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.492058    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"]
Jul 24 18:19:45.500467 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.500448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:45.530076 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.530053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-credential-socket\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.530172 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.530079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-envoy\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.530172 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.530100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg24t\" (UniqueName: \"kubernetes.io/projected/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-kube-api-access-dg24t\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.530172 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.530130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-data\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.530271 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.530176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-workload-socket\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.530271 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.530237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istiod-ca-cert\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.530271 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.530265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-podinfo\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.530363 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.530301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-workload-certs\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.530363 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.530319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-token\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.631671 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.631641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-data\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.631834 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.631692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-workload-socket\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.631834 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.631729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istiod-ca-cert\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.631834 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.631749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-podinfo\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.631834 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.631781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-workload-certs\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.631834 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.631799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-token\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.631834 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.631831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-credential-socket\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.632158 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.631984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-envoy\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.632158 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.632046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg24t\" (UniqueName: \"kubernetes.io/projected/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-kube-api-access-dg24t\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.632158 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.632077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-data\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.632380 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.632333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-workload-socket\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.632433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.632380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-workload-certs\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.632433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.632412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-credential-socket\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.632513 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.632492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istiod-ca-cert\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.634210 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.634188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-envoy\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.634315 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.634298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-podinfo\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.640304 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.640274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-token\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.640447 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.640433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg24t\" (UniqueName: \"kubernetes.io/projected/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-kube-api-access-dg24t\") pod \"e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") " pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.791424 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.791338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:45.912604 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.911090    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:45.961599 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.953659    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"]
Jul 24 18:19:45.972886 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.972846    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:19:45.973019 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.972932    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:19:45.973019 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:45.972973    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:19:46.404441 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:46.404398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j" event={"ID":"2ce832ed-08d8-4587-88a8-c6e9ce5af9be","Type":"ContainerStarted","Data":"4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166"}
Jul 24 18:19:46.404441 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:46.404449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j" event={"ID":"2ce832ed-08d8-4587-88a8-c6e9ce5af9be","Type":"ContainerStarted","Data":"00ace0896ddb8cd6d1b5eb9110de1975d4779b6c39da20f1fcf990024965be99"}
Jul 24 18:19:46.427693 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:46.427645    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j" podStartSLOduration=1.427630202 podStartE2EDuration="1.427630202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:19:46.427222567 +0000 UTC m=+2599.943733604" watchObservedRunningTime="2026-07-24 18:19:46.427630202 +0000 UTC m=+2599.944141238"
Jul 24 18:19:46.791617 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:46.791526    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:47.796755 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:47.796727    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:48.371777 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.371747    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:48.395261 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.395235    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:19:48.417105 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.417075    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:48.418014 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.417994    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:19:48.470544 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.470510    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"]
Jul 24 18:19:48.476958 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.476931    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc04ac9a-531b-482b-af64-45306f05eb0a" containerName="istio-proxy"
Jul 24 18:19:48.478615 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.476266    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:48.482611 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.480986    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:48.484670 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.484650    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c4d732-serving-cert\""
Jul 24 18:19:48.486763 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.486739    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"]
Jul 24 18:19:48.563913 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.563785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602b34f9-782d-4eb8-9a10-51e824605d0a-maas-api-tls\") pod \"maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j\" (UID: \"602b34f9-782d-4eb8-9a10-51e824605d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:48.563913 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.563843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602b34f9-782d-4eb8-9a10-51e824605d0a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j\" (UID: \"602b34f9-782d-4eb8-9a10-51e824605d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:48.564178 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.563971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ng8h\" (UniqueName: \"kubernetes.io/projected/602b34f9-782d-4eb8-9a10-51e824605d0a-kube-api-access-6ng8h\") pod \"maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j\" (UID: \"602b34f9-782d-4eb8-9a10-51e824605d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:48.564806 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.564779    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:48.660471 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.660375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:48.665176 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.665144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ng8h\" (UniqueName: \"kubernetes.io/projected/602b34f9-782d-4eb8-9a10-51e824605d0a-kube-api-access-6ng8h\") pod \"maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j\" (UID: \"602b34f9-782d-4eb8-9a10-51e824605d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:48.665314 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.665240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602b34f9-782d-4eb8-9a10-51e824605d0a-maas-api-tls\") pod \"maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j\" (UID: \"602b34f9-782d-4eb8-9a10-51e824605d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:48.665314 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.665272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602b34f9-782d-4eb8-9a10-51e824605d0a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j\" (UID: \"602b34f9-782d-4eb8-9a10-51e824605d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:48.666146 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.666111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602b34f9-782d-4eb8-9a10-51e824605d0a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j\" (UID: \"602b34f9-782d-4eb8-9a10-51e824605d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:48.668969 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.668947    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602b34f9-782d-4eb8-9a10-51e824605d0a-maas-api-tls\") pod \"maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j\" (UID: \"602b34f9-782d-4eb8-9a10-51e824605d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:48.675201 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.675181    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ng8h\" (UniqueName: \"kubernetes.io/projected/602b34f9-782d-4eb8-9a10-51e824605d0a-kube-api-access-6ng8h\") pod \"maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j\" (UID: \"602b34f9-782d-4eb8-9a10-51e824605d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:48.795457 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:48.795420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:49.162594 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.162217    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"]
Jul 24 18:19:49.176728 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:19:49.176696    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602b34f9_782d_4eb8_9a10_51e824605d0a.slice/crio-65d6d32d57335eca02f84e47553a8483b19eb923e41b944a7eba68de97aa1954 WatchSource:0}: Error finding container 65d6d32d57335eca02f84e47553a8483b19eb923e41b944a7eba68de97aa1954: Status 404 returned error can't find the container with id 65d6d32d57335eca02f84e47553a8483b19eb923e41b944a7eba68de97aa1954
Jul 24 18:19:49.205479 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.205451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:49.303912 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.303880    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:49.385686 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.385652    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"]
Jul 24 18:19:49.390068 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.390051    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"
Jul 24 18:19:49.392657 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.392639    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-c4d732\""
Jul 24 18:19:49.392860 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.392848    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-c4d732-dockercfg-ls64d\""
Jul 24 18:19:49.399198 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.399177    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"]
Jul 24 18:19:49.422545 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.422454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j" event={"ID":"602b34f9-782d-4eb8-9a10-51e824605d0a","Type":"ContainerStarted","Data":"65d6d32d57335eca02f84e47553a8483b19eb923e41b944a7eba68de97aa1954"}
Jul 24 18:19:49.478129 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.478097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psx9n\" (UniqueName: \"kubernetes.io/projected/ceb1d686-0057-4b5e-85a1-8672e91c6e54-kube-api-access-psx9n\") pod \"payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg\" (UID: \"ceb1d686-0057-4b5e-85a1-8672e91c6e54\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"
Jul 24 18:19:49.478478 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.478455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb1d686-0057-4b5e-85a1-8672e91c6e54-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg\" (UID: \"ceb1d686-0057-4b5e-85a1-8672e91c6e54\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"
Jul 24 18:19:49.579954 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.579923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb1d686-0057-4b5e-85a1-8672e91c6e54-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg\" (UID: \"ceb1d686-0057-4b5e-85a1-8672e91c6e54\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"
Jul 24 18:19:49.580073 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.580004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psx9n\" (UniqueName: \"kubernetes.io/projected/ceb1d686-0057-4b5e-85a1-8672e91c6e54-kube-api-access-psx9n\") pod \"payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg\" (UID: \"ceb1d686-0057-4b5e-85a1-8672e91c6e54\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"
Jul 24 18:19:49.580617 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.580597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb1d686-0057-4b5e-85a1-8672e91c6e54-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg\" (UID: \"ceb1d686-0057-4b5e-85a1-8672e91c6e54\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"
Jul 24 18:19:49.588224 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.588200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psx9n\" (UniqueName: \"kubernetes.io/projected/ceb1d686-0057-4b5e-85a1-8672e91c6e54-kube-api-access-psx9n\") pod \"payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg\" (UID: \"ceb1d686-0057-4b5e-85a1-8672e91c6e54\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"
Jul 24 18:19:49.701193 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.701117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"
Jul 24 18:19:49.872817 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:49.872680    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"]
Jul 24 18:19:49.877154 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:19:49.877129    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb1d686_0057_4b5e_85a1_8672e91c6e54.slice/crio-bb6162c7c14605e37a93df48c626b78bde46d315266c15c813c9ed7bd795f3a3 WatchSource:0}: Error finding container bb6162c7c14605e37a93df48c626b78bde46d315266c15c813c9ed7bd795f3a3: Status 404 returned error can't find the container with id bb6162c7c14605e37a93df48c626b78bde46d315266c15c813c9ed7bd795f3a3
Jul 24 18:19:50.428820 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:50.428781    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j" event={"ID":"602b34f9-782d-4eb8-9a10-51e824605d0a","Type":"ContainerStarted","Data":"f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb"}
Jul 24 18:19:50.429235 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:50.428893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:50.430325 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:50.430292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg" event={"ID":"ceb1d686-0057-4b5e-85a1-8672e91c6e54","Type":"ContainerStarted","Data":"c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8"}
Jul 24 18:19:50.430451 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:50.430342    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg" event={"ID":"ceb1d686-0057-4b5e-85a1-8672e91c6e54","Type":"ContainerStarted","Data":"bb6162c7c14605e37a93df48c626b78bde46d315266c15c813c9ed7bd795f3a3"}
Jul 24 18:19:50.430497 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:50.430481    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"
Jul 24 18:19:50.449200 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:50.449140    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j" podStartSLOduration=2.078072737 podStartE2EDuration="2.449120066s" totalImagesPullingTime="371.047329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:19:50.447158698 +0000 UTC m=+2603.963669729" watchObservedRunningTime="2026-07-24 18:19:50.449120066 +0000 UTC m=+2603.965631108"
Jul 24 18:19:50.463363 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:50.463216    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg" podStartSLOduration=1.463196658 podStartE2EDuration="1.463196658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:19:50.46264644 +0000 UTC m=+2603.979157478" watchObservedRunningTime="2026-07-24 18:19:50.463196658 +0000 UTC m=+2603.979707696"
Jul 24 18:19:52.886757 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:52.886724    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:19:53.390593 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:53.390530    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb" podUID="f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.62:9004: connect: connection refused"
Jul 24 18:19:53.391044 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:53.391023    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"
Jul 24 18:19:53.449916 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:53.449887    2581 generic.go:356] "Generic (PLEG): container finished" podID="233b69e0-5dd4-4235-a7f4-3bff6c32f89c" containerID="942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5" exitCode=0
Jul 24 18:19:53.450081 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:53.449931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb" event={"ID":"233b69e0-5dd4-4235-a7f4-3bff6c32f89c","Type":"ContainerDied","Data":"942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5"}
Jul 24 18:19:53.450377 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:53.450360    2581 scope.go:117] "RemoveContainer" containerID="942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5"
Jul 24 18:19:53.451540 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:53.451507    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:19:53.451637 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:53.451620    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:19:53.451679 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:53.451669    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:19:54.364560 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:54.364492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:54.459107 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:54.459061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb" event={"ID":"233b69e0-5dd4-4235-a7f4-3bff6c32f89c","Type":"ContainerStarted","Data":"27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440"}
Jul 24 18:19:54.517468 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:54.517439    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:54.518406 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:54.518378    2581 patch_prober.go:28] interesting pod/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 24 18:19:54.518457 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:54.518428    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb" podUID="233b69e0-5dd4-4235-a7f4-3bff6c32f89c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 24 18:19:55.518243 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:55.518204    2581 patch_prober.go:28] interesting pod/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 24 18:19:55.518664 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:55.518258    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb" podUID="233b69e0-5dd4-4235-a7f4-3bff6c32f89c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 24 18:19:56.455835 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:56.455802    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:19:56.518212 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:56.518167    2581 patch_prober.go:28] interesting pod/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 24 18:19:56.518368 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:56.518241    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb" podUID="233b69e0-5dd4-4235-a7f4-3bff6c32f89c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 24 18:19:57.517897 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:57.517850    2581 patch_prober.go:28] interesting pod/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 24 18:19:57.518064 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:57.517919    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb" podUID="233b69e0-5dd4-4235-a7f4-3bff6c32f89c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 24 18:19:58.522094 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:58.522062    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:58.522469 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:58.522340    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:19:58.522967 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:19:58.522943    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:20:01.439891 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:01.439857    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"
Jul 24 18:20:02.508499 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:02.508460    2581 generic.go:356] "Generic (PLEG): container finished" podID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerID="4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166" exitCode=0
Jul 24 18:20:02.508937 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:02.508535    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j" event={"ID":"2ce832ed-08d8-4587-88a8-c6e9ce5af9be","Type":"ContainerDied","Data":"4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166"}
Jul 24 18:20:02.509006 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:02.508990    2581 scope.go:117] "RemoveContainer" containerID="4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166"
Jul 24 18:20:02.509986 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:02.509964    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:20:02.510046 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:02.510033    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:20:02.510087 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:02.510068    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 18:20:03.390187 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:03.390155    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"
Jul 24 18:20:03.417126 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:03.417101    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:20:03.515503 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:03.515464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j" event={"ID":"2ce832ed-08d8-4587-88a8-c6e9ce5af9be","Type":"ContainerStarted","Data":"c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f"}
Jul 24 18:20:03.792336 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:03.792234    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:20:03.793531 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:03.793501    2581 patch_prober.go:28] interesting pod/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 24 18:20:03.793691 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:03.793561    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j" podUID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 24 18:20:04.792072 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:04.792040    2581 patch_prober.go:28] interesting pod/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 24 18:20:04.792507 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:04.792099    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j" podUID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 24 18:20:05.792007 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:05.791971    2581 patch_prober.go:28] interesting pod/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 24 18:20:05.792186 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:05.792025    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j" podUID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 24 18:20:06.792323 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:06.792288    2581 patch_prober.go:28] interesting pod/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 24 18:20:06.792722 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:06.792336    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j" podUID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 24 18:20:06.988781 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:06.988752    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:07.795239 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:07.795208    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:20:07.795730 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:07.795544    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:20:07.796267 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:07.796245    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:20:24.411982 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:24.411943    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:27.175765 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:27.175735    2581 scope.go:117] "RemoveContainer" containerID="fd7d34cc529d77b4f82d48e176b5e5417dd80881c7369a44a76df660af2e12ed"
Jul 24 18:20:36.763988 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:36.763950    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:53.612115 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:53.612076    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:20:53.670955 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:20:53.670920    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:22:29.049765 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:22:29.049716    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:22:29.136146 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:22:29.136118    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:22:56.786897 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:22:56.786809    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:22:57.308828 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:22:57.308792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:22:57.757430 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:22:57.757333    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:23:14.188459 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:23:14.188427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:24:06.788390 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:24:06.788363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:24:10.594041 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:24:10.594005    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:24:42.169627 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:24:42.169591    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:24:43.082287 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:24:43.082256    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:03.809928 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:25:03.809894    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:25:24.848011 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:25:24.847975    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:26:10.460720 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:26:10.460640    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:26:11.457128 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:26:11.457089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:26:30.792592 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:26:30.792534    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:26:31.136155 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:26:31.136114    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:26:31.498395 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:26:31.498305    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:26:42.715098 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:26:42.715054    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:00.263732 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:00.263689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:00.356930 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:00.356891    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:00.503808 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:00.503773    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:00.848203 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:00.848170    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:01.235197 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.235110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:01.330699 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.330659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:01.449130 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.449097    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"]
Jul 24 18:27:01.452300 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.452284    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"
Jul 24 18:27:01.455901 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.455879    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7ee2e6\""
Jul 24 18:27:01.456009 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.455918    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7ee2e6-dockercfg-gvg4k\""
Jul 24 18:27:01.495118 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.495031    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"]
Jul 24 18:27:01.519095 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.519055    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"]
Jul 24 18:27:01.522357 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.522335    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"
Jul 24 18:27:01.547970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.547944    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"]
Jul 24 18:27:01.573478 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.573455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf\" (UID: \"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"
Jul 24 18:27:01.573632 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.573503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bppl2\" (UniqueName: \"kubernetes.io/projected/d69b6d93-5170-40ca-9920-6f5e33c2aa21-kube-api-access-bppl2\") pod \"payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7\" (UID: \"d69b6d93-5170-40ca-9920-6f5e33c2aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"
Jul 24 18:27:01.573632 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.573527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69b6d93-5170-40ca-9920-6f5e33c2aa21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7\" (UID: \"d69b6d93-5170-40ca-9920-6f5e33c2aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"
Jul 24 18:27:01.573632 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.573606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7llsj\" (UniqueName: \"kubernetes.io/projected/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc-kube-api-access-7llsj\") pod \"payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf\" (UID: \"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"
Jul 24 18:27:01.674943 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.674907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf\" (UID: \"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"
Jul 24 18:27:01.675117 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.674958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bppl2\" (UniqueName: \"kubernetes.io/projected/d69b6d93-5170-40ca-9920-6f5e33c2aa21-kube-api-access-bppl2\") pod \"payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7\" (UID: \"d69b6d93-5170-40ca-9920-6f5e33c2aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"
Jul 24 18:27:01.675117 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.675068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69b6d93-5170-40ca-9920-6f5e33c2aa21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7\" (UID: \"d69b6d93-5170-40ca-9920-6f5e33c2aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"
Jul 24 18:27:01.675248 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.675116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llsj\" (UniqueName: \"kubernetes.io/projected/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc-kube-api-access-7llsj\") pod \"payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf\" (UID: \"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"
Jul 24 18:27:01.675644 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.675623    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf\" (UID: \"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"
Jul 24 18:27:01.675818 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.675797    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69b6d93-5170-40ca-9920-6f5e33c2aa21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7\" (UID: \"d69b6d93-5170-40ca-9920-6f5e33c2aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"
Jul 24 18:27:01.683941 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.683916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bppl2\" (UniqueName: \"kubernetes.io/projected/d69b6d93-5170-40ca-9920-6f5e33c2aa21-kube-api-access-bppl2\") pod \"payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7\" (UID: \"d69b6d93-5170-40ca-9920-6f5e33c2aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"
Jul 24 18:27:01.685230 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.685213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llsj\" (UniqueName: \"kubernetes.io/projected/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc-kube-api-access-7llsj\") pod \"payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf\" (UID: \"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"
Jul 24 18:27:01.762625 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.762528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"
Jul 24 18:27:01.833246 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.833028    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"
Jul 24 18:27:01.934372 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:01.934171    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"]
Jul 24 18:27:01.936602 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:27:01.936541    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69b6d93_5170_40ca_9920_6f5e33c2aa21.slice/crio-e11517b24976b0815d8a90f944a9926f19a6b0a26b601c13f6cfd67ef35f2305 WatchSource:0}: Error finding container e11517b24976b0815d8a90f944a9926f19a6b0a26b601c13f6cfd67ef35f2305: Status 404 returned error can't find the container with id e11517b24976b0815d8a90f944a9926f19a6b0a26b601c13f6cfd67ef35f2305
Jul 24 18:27:02.012525 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:02.012503    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"]
Jul 24 18:27:02.015090 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:27:02.015067    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f4a2ef_d9ea_4c0c_9395_a7673bef32fc.slice/crio-43fbcd52f23de75f8741c6cf741bef46004ece334c6af901b7ac8b93e72ce4bf WatchSource:0}: Error finding container 43fbcd52f23de75f8741c6cf741bef46004ece334c6af901b7ac8b93e72ce4bf: Status 404 returned error can't find the container with id 43fbcd52f23de75f8741c6cf741bef46004ece334c6af901b7ac8b93e72ce4bf
Jul 24 18:27:02.362967 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:02.362929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf" event={"ID":"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc","Type":"ContainerStarted","Data":"82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47"}
Jul 24 18:27:02.363427 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:02.362985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf" event={"ID":"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc","Type":"ContainerStarted","Data":"43fbcd52f23de75f8741c6cf741bef46004ece334c6af901b7ac8b93e72ce4bf"}
Jul 24 18:27:02.363427 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:02.363015    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"
Jul 24 18:27:02.364476 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:02.364452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7" event={"ID":"d69b6d93-5170-40ca-9920-6f5e33c2aa21","Type":"ContainerStarted","Data":"58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6"}
Jul 24 18:27:02.364610 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:02.364495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7" event={"ID":"d69b6d93-5170-40ca-9920-6f5e33c2aa21","Type":"ContainerStarted","Data":"e11517b24976b0815d8a90f944a9926f19a6b0a26b601c13f6cfd67ef35f2305"}
Jul 24 18:27:02.364610 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:02.364513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"
Jul 24 18:27:02.412469 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:02.412422    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf" podStartSLOduration=1.412407592 podStartE2EDuration="1.412407592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:27:02.409938619 +0000 UTC m=+3035.926449649" watchObservedRunningTime="2026-07-24 18:27:02.412407592 +0000 UTC m=+3035.928918627"
Jul 24 18:27:02.451185 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:02.451127    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7" podStartSLOduration=1.45110634 podStartE2EDuration="1.45110634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:27:02.448717179 +0000 UTC m=+3035.965228217" watchObservedRunningTime="2026-07-24 18:27:02.45110634 +0000 UTC m=+3035.967617382"
Jul 24 18:27:04.783937 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:04.783905    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:13.371487 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:13.371461    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"
Jul 24 18:27:13.371897 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:13.371529    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"
Jul 24 18:27:15.845182 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:15.845148    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:16.121229 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:16.121147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:16.318081 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:16.318043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:18.738845 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:18.738810    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:18.817492 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:18.817461    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:18.822099 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:18.822075    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"]
Jul 24 18:27:18.825455 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:18.825435    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:18.827827 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:18.827802    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e18e67-serving-cert\""
Jul 24 18:27:18.837714 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:18.837694    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"]
Jul 24 18:27:18.918993 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:18.918930    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:18.920037 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:18.920019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:18.920095 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:18.920063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:18.920095 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:18.920082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l22pd\" (UniqueName: \"kubernetes.io/projected/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-kube-api-access-l22pd\") pod \"maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:19.021138 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.021057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:19.021138 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.021089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22pd\" (UniqueName: \"kubernetes.io/projected/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-kube-api-access-l22pd\") pod \"maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:19.021375 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.021144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:19.021375 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:27:19.021249    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-serving-cert: secret "maas-api-e2e-ipp-b-e18e67-serving-cert" not found
Jul 24 18:27:19.021375 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:27:19.021298    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-maas-api-tls podName:4ee62993-dcdb-4fee-a3b3-afeb1a3ab364 nodeName:}" failed. No retries permitted until 2026-07-24 18:27:19.521284099 +0000 UTC m=+3053.037795113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-maas-api-tls") pod "maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2" (UID: "4ee62993-dcdb-4fee-a3b3-afeb1a3ab364") : secret "maas-api-e2e-ipp-b-e18e67-serving-cert" not found
Jul 24 18:27:19.021723 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.021705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:19.034808 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.034785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l22pd\" (UniqueName: \"kubernetes.io/projected/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-kube-api-access-l22pd\") pod \"maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:19.193784 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.193751    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:19.520325 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.520292    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:19.525508 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.525486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:19.527916 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.527893    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:19.620863 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.620826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:19.737260 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.737228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:19.887902 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.887835    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"]
Jul 24 18:27:19.890386 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:27:19.890353    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee62993_dcdb_4fee_a3b3_afeb1a3ab364.slice/crio-93b799f33e9682b2e78f6a3cbc434e34a8d9ffea982dee54e1b608ff721bb690 WatchSource:0}: Error finding container 93b799f33e9682b2e78f6a3cbc434e34a8d9ffea982dee54e1b608ff721bb690: Status 404 returned error can't find the container with id 93b799f33e9682b2e78f6a3cbc434e34a8d9ffea982dee54e1b608ff721bb690
Jul 24 18:27:19.891874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:19.891855    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:27:20.446677 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:20.446641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2" event={"ID":"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364","Type":"ContainerStarted","Data":"633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c"}
Jul 24 18:27:20.446836 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:20.446694    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2" event={"ID":"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364","Type":"ContainerStarted","Data":"93b799f33e9682b2e78f6a3cbc434e34a8d9ffea982dee54e1b608ff721bb690"}
Jul 24 18:27:20.446836 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:20.446726    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:20.464944 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:20.464900    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2" podStartSLOduration=2.089937321 podStartE2EDuration="2.464884869s" totalImagesPullingTime="374.947548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:27:20.463500077 +0000 UTC m=+3053.980011114" watchObservedRunningTime="2026-07-24 18:27:20.464884869 +0000 UTC m=+3053.981395905"
Jul 24 18:27:23.250901 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:23.250859    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:26.456773 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:26.456743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:27:51.026523 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:51.026491    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:27:57.898213 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:27:57.898176    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:04.917172 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:04.917140    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:04.974771 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:04.974732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:22.615769 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:22.615743    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf_68f4a2ef-d9ea-4c0c-9395-a7673bef32fc/payload-processing/0.log"
Jul 24 18:28:23.155932 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:23.155897    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:23.477013 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:23.476926    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:23.839055 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:23.839018    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:26.632118 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.632080    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:26.725205 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.725175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:26.848101 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.848066    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"]
Jul 24 18:28:26.851486 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.851468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:26.854200 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.854180    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-a2a4d8-serving-cert\""
Jul 24 18:28:26.881115 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.881090    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:26.882255 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.882204    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"]
Jul 24 18:28:26.891422 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.891397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118dbac4-656e-4bdb-954f-dbea4fdc6c78-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:26.891535 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.891441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118dbac4-656e-4bdb-954f-dbea4fdc6c78-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:26.891623 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.891566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgql\" (UniqueName: \"kubernetes.io/projected/118dbac4-656e-4bdb-954f-dbea4fdc6c78-kube-api-access-8hgql\") pod \"maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:26.992651 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.992618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118dbac4-656e-4bdb-954f-dbea4fdc6c78-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:26.992835 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.992662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118dbac4-656e-4bdb-954f-dbea4fdc6c78-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:26.992835 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.992784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgql\" (UniqueName: \"kubernetes.io/projected/118dbac4-656e-4bdb-954f-dbea4fdc6c78-kube-api-access-8hgql\") pod \"maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:26.992965 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:28:26.992852    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-serving-cert: secret "maas-api-e2e-ipp-cleanup-a2a4d8-serving-cert" not found
Jul 24 18:28:26.992965 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:28:26.992930    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/118dbac4-656e-4bdb-954f-dbea4fdc6c78-maas-api-tls podName:118dbac4-656e-4bdb-954f-dbea4fdc6c78 nodeName:}" failed. No retries permitted until 2026-07-24 18:28:27.492908543 +0000 UTC m=+3121.009419571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/118dbac4-656e-4bdb-954f-dbea4fdc6c78-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs" (UID: "118dbac4-656e-4bdb-954f-dbea4fdc6c78") : secret "maas-api-e2e-ipp-cleanup-a2a4d8-serving-cert" not found
Jul 24 18:28:26.993315 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:26.993297    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118dbac4-656e-4bdb-954f-dbea4fdc6c78-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:27.012672 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.012650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgql\" (UniqueName: \"kubernetes.io/projected/118dbac4-656e-4bdb-954f-dbea4fdc6c78-kube-api-access-8hgql\") pod \"maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:27.132140 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.132113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:27.419667 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.419607    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:27.497177 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.497144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118dbac4-656e-4bdb-954f-dbea4fdc6c78-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:27.499526 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.499499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118dbac4-656e-4bdb-954f-dbea4fdc6c78-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:27.519490 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.519466    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:27.560077 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.560052    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"]
Jul 24 18:28:27.563544 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.563528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"
Jul 24 18:28:27.566222 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.566204    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-a2a4d8-dockercfg-46mn5\""
Jul 24 18:28:27.566345 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.566218    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-a2a4d8\""
Jul 24 18:28:27.574846 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.574822    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"]
Jul 24 18:28:27.597636 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.597614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44c97\" (UniqueName: \"kubernetes.io/projected/8e76a999-eba4-4b65-8251-afc4e6821cac-kube-api-access-44c97\") pod \"payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq\" (UID: \"8e76a999-eba4-4b65-8251-afc4e6821cac\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"
Jul 24 18:28:27.597763 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.597693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e76a999-eba4-4b65-8251-afc4e6821cac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq\" (UID: \"8e76a999-eba4-4b65-8251-afc4e6821cac\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"
Jul 24 18:28:27.698746 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.698668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e76a999-eba4-4b65-8251-afc4e6821cac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq\" (UID: \"8e76a999-eba4-4b65-8251-afc4e6821cac\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"
Jul 24 18:28:27.699122 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.698760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44c97\" (UniqueName: \"kubernetes.io/projected/8e76a999-eba4-4b65-8251-afc4e6821cac-kube-api-access-44c97\") pod \"payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq\" (UID: \"8e76a999-eba4-4b65-8251-afc4e6821cac\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"
Jul 24 18:28:27.699363 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.699342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e76a999-eba4-4b65-8251-afc4e6821cac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq\" (UID: \"8e76a999-eba4-4b65-8251-afc4e6821cac\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"
Jul 24 18:28:27.708857 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.708835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44c97\" (UniqueName: \"kubernetes.io/projected/8e76a999-eba4-4b65-8251-afc4e6821cac-kube-api-access-44c97\") pod \"payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq\" (UID: \"8e76a999-eba4-4b65-8251-afc4e6821cac\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"
Jul 24 18:28:27.761295 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.761269    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:27.875768 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.875738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"
Jul 24 18:28:27.904192 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:27.904169    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"]
Jul 24 18:28:27.906564 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:28:27.906538    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118dbac4_656e_4bdb_954f_dbea4fdc6c78.slice/crio-bb6e23dacbd90964c346133fff4ecb25b48e26c118100b92d18256c45e5b12e0 WatchSource:0}: Error finding container bb6e23dacbd90964c346133fff4ecb25b48e26c118100b92d18256c45e5b12e0: Status 404 returned error can't find the container with id bb6e23dacbd90964c346133fff4ecb25b48e26c118100b92d18256c45e5b12e0
Jul 24 18:28:28.011791 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:28.011760    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"]
Jul 24 18:28:28.013723 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:28:28.013694    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e76a999_eba4_4b65_8251_afc4e6821cac.slice/crio-8661168ba35f36b56db2e44efccc3b755789983cef87711a5335bf023cddda5b WatchSource:0}: Error finding container 8661168ba35f36b56db2e44efccc3b755789983cef87711a5335bf023cddda5b: Status 404 returned error can't find the container with id 8661168ba35f36b56db2e44efccc3b755789983cef87711a5335bf023cddda5b
Jul 24 18:28:28.748728 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:28.748529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq" event={"ID":"8e76a999-eba4-4b65-8251-afc4e6821cac","Type":"ContainerStarted","Data":"98536c40b7a433d2181965ec789c3cbe9c21d0fc7ca4a9f2368f9b358fe15a72"}
Jul 24 18:28:28.748728 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:28.748622    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"
Jul 24 18:28:28.748728 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:28.748645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq" event={"ID":"8e76a999-eba4-4b65-8251-afc4e6821cac","Type":"ContainerStarted","Data":"8661168ba35f36b56db2e44efccc3b755789983cef87711a5335bf023cddda5b"}
Jul 24 18:28:28.750159 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:28.750133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs" event={"ID":"118dbac4-656e-4bdb-954f-dbea4fdc6c78","Type":"ContainerStarted","Data":"603cc7bb62798ec78dfc40c739e6b8d2716bd9bee607602d6ae2af19e1ca9549"}
Jul 24 18:28:28.750294 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:28.750175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs" event={"ID":"118dbac4-656e-4bdb-954f-dbea4fdc6c78","Type":"ContainerStarted","Data":"bb6e23dacbd90964c346133fff4ecb25b48e26c118100b92d18256c45e5b12e0"}
Jul 24 18:28:28.750294 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:28.750277    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:28.770663 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:28.770619    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq" podStartSLOduration=1.770604923 podStartE2EDuration="1.770604923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:28:28.768827638 +0000 UTC m=+3122.285338675" watchObservedRunningTime="2026-07-24 18:28:28.770604923 +0000 UTC m=+3122.287115961"
Jul 24 18:28:28.789023 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:28.788984    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs" podStartSLOduration=2.385859863 podStartE2EDuration="2.78897027s" totalImagesPullingTime="403.110407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:28:28.78701317 +0000 UTC m=+3122.303524205" watchObservedRunningTime="2026-07-24 18:28:28.78897027 +0000 UTC m=+3122.305481305"
Jul 24 18:28:31.151104 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:31.151070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:34.759781 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:34.759754    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:39.757131 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:39.757102    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"
Jul 24 18:28:52.674240 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:52.674157    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"]
Jul 24 18:28:52.674809 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:52.674379    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs" podUID="118dbac4-656e-4bdb-954f-dbea4fdc6c78" containerName="maas-api" containerID="cri-o://603cc7bb62798ec78dfc40c739e6b8d2716bd9bee607602d6ae2af19e1ca9549" gracePeriod=30
Jul 24 18:28:52.714249 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:52.714226    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:52.743152 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:52.743121    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"]
Jul 24 18:28:52.743416 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:52.743363    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq" podUID="8e76a999-eba4-4b65-8251-afc4e6821cac" containerName="payload-processing" containerID="cri-o://98536c40b7a433d2181965ec789c3cbe9c21d0fc7ca4a9f2368f9b358fe15a72" gracePeriod=30
Jul 24 18:28:52.818837 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:52.818806    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:52.868480 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:52.868449    2581 generic.go:356] "Generic (PLEG): container finished" podID="118dbac4-656e-4bdb-954f-dbea4fdc6c78" containerID="603cc7bb62798ec78dfc40c739e6b8d2716bd9bee607602d6ae2af19e1ca9549" exitCode=0
Jul 24 18:28:52.868620 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:52.868529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs" event={"ID":"118dbac4-656e-4bdb-954f-dbea4fdc6c78","Type":"ContainerDied","Data":"603cc7bb62798ec78dfc40c739e6b8d2716bd9bee607602d6ae2af19e1ca9549"}
Jul 24 18:28:52.870319 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:52.870284    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e76a999-eba4-4b65-8251-afc4e6821cac" containerID="98536c40b7a433d2181965ec789c3cbe9c21d0fc7ca4a9f2368f9b358fe15a72" exitCode=0
Jul 24 18:28:52.870422 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:52.870357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq" event={"ID":"8e76a999-eba4-4b65-8251-afc4e6821cac","Type":"ContainerDied","Data":"98536c40b7a433d2181965ec789c3cbe9c21d0fc7ca4a9f2368f9b358fe15a72"}
Jul 24 18:28:52.934656 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:52.931208    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:52.996900 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:52.996878    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:53.028597 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.028557    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"
Jul 24 18:28:53.132001 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.131973    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118dbac4-656e-4bdb-954f-dbea4fdc6c78-maas-api-tls\") pod \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") "
Jul 24 18:28:53.132157 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.132047    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118dbac4-656e-4bdb-954f-dbea4fdc6c78-openshift-service-ca\") pod \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") "
Jul 24 18:28:53.132157 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.132093    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hgql\" (UniqueName: \"kubernetes.io/projected/118dbac4-656e-4bdb-954f-dbea4fdc6c78-kube-api-access-8hgql\") pod \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\" (UID: \"118dbac4-656e-4bdb-954f-dbea4fdc6c78\") "
Jul 24 18:28:53.132157 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.132120    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44c97\" (UniqueName: \"kubernetes.io/projected/8e76a999-eba4-4b65-8251-afc4e6821cac-kube-api-access-44c97\") pod \"8e76a999-eba4-4b65-8251-afc4e6821cac\" (UID: \"8e76a999-eba4-4b65-8251-afc4e6821cac\") "
Jul 24 18:28:53.132157 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.132148    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e76a999-eba4-4b65-8251-afc4e6821cac-plugins-config-volume\") pod \"8e76a999-eba4-4b65-8251-afc4e6821cac\" (UID: \"8e76a999-eba4-4b65-8251-afc4e6821cac\") "
Jul 24 18:28:53.132406 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.132379    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118dbac4-656e-4bdb-954f-dbea4fdc6c78-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "118dbac4-656e-4bdb-954f-dbea4fdc6c78" (UID: "118dbac4-656e-4bdb-954f-dbea4fdc6c78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:28:53.132637 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.132611    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e76a999-eba4-4b65-8251-afc4e6821cac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e76a999-eba4-4b65-8251-afc4e6821cac" (UID: "8e76a999-eba4-4b65-8251-afc4e6821cac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:28:53.134041 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.134023    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118dbac4-656e-4bdb-954f-dbea4fdc6c78-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "118dbac4-656e-4bdb-954f-dbea4fdc6c78" (UID: "118dbac4-656e-4bdb-954f-dbea4fdc6c78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:28:53.134157 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.134142    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118dbac4-656e-4bdb-954f-dbea4fdc6c78-kube-api-access-8hgql" (OuterVolumeSpecName: "kube-api-access-8hgql") pod "118dbac4-656e-4bdb-954f-dbea4fdc6c78" (UID: "118dbac4-656e-4bdb-954f-dbea4fdc6c78"). InnerVolumeSpecName "kube-api-access-8hgql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:28:53.134215 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.134192    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e76a999-eba4-4b65-8251-afc4e6821cac-kube-api-access-44c97" (OuterVolumeSpecName: "kube-api-access-44c97") pod "8e76a999-eba4-4b65-8251-afc4e6821cac" (UID: "8e76a999-eba4-4b65-8251-afc4e6821cac"). InnerVolumeSpecName "kube-api-access-44c97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:28:53.160641 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.160618    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:53.219902 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.219841    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:53.233029 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.233003    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118dbac4-656e-4bdb-954f-dbea4fdc6c78-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:28:53.233029 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.233025    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hgql\" (UniqueName: \"kubernetes.io/projected/118dbac4-656e-4bdb-954f-dbea4fdc6c78-kube-api-access-8hgql\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:28:53.233159 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.233042    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44c97\" (UniqueName: \"kubernetes.io/projected/8e76a999-eba4-4b65-8251-afc4e6821cac-kube-api-access-44c97\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:28:53.233159 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.233055    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e76a999-eba4-4b65-8251-afc4e6821cac-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:28:53.233159 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.233068    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118dbac4-656e-4bdb-954f-dbea4fdc6c78-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:28:53.876215 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.876188    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"
Jul 24 18:28:53.876692 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.876234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq" event={"ID":"8e76a999-eba4-4b65-8251-afc4e6821cac","Type":"ContainerDied","Data":"8661168ba35f36b56db2e44efccc3b755789983cef87711a5335bf023cddda5b"}
Jul 24 18:28:53.876692 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.876300    2581 scope.go:117] "RemoveContainer" containerID="98536c40b7a433d2181965ec789c3cbe9c21d0fc7ca4a9f2368f9b358fe15a72"
Jul 24 18:28:53.880694 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.880530    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"
Jul 24 18:28:53.880694 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.880531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs" event={"ID":"118dbac4-656e-4bdb-954f-dbea4fdc6c78","Type":"ContainerDied","Data":"bb6e23dacbd90964c346133fff4ecb25b48e26c118100b92d18256c45e5b12e0"}
Jul 24 18:28:53.888628 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.888559    2581 scope.go:117] "RemoveContainer" containerID="603cc7bb62798ec78dfc40c739e6b8d2716bd9bee607602d6ae2af19e1ca9549"
Jul 24 18:28:53.899356 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.899335    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"]
Jul 24 18:28:53.903362 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.903342    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a2a4d8-554b689fcb-wrqrq"]
Jul 24 18:28:53.913366 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.913340    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"]
Jul 24 18:28:53.917138 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:53.917119    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a2a4d8-5c69b847b-r9vrs"]
Jul 24 18:28:54.963446 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:54.963415    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="118dbac4-656e-4bdb-954f-dbea4fdc6c78" path="/var/lib/kubelet/pods/118dbac4-656e-4bdb-954f-dbea4fdc6c78/volumes"
Jul 24 18:28:54.963842 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:54.963773    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e76a999-eba4-4b65-8251-afc4e6821cac" path="/var/lib/kubelet/pods/8e76a999-eba4-4b65-8251-afc4e6821cac/volumes"
Jul 24 18:28:57.124376 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:57.124351    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:28:57.497276 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:28:57.497187    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:03.518956 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:03.518921    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:03.614428 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:03.614393    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:03.793411 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:03.793317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:03.999876 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:03.999842    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:22.904485 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:22.904445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:22.926974 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:22.926934    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"]
Jul 24 18:29:22.927220 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:22.927195    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf" podUID="68f4a2ef-d9ea-4c0c-9395-a7673bef32fc" containerName="payload-processing" containerID="cri-o://82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47" gracePeriod=30
Jul 24 18:29:22.936787 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:22.936476    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"]
Jul 24 18:29:22.936787 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:22.936711    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7" podUID="d69b6d93-5170-40ca-9920-6f5e33c2aa21" containerName="payload-pre-processing" containerID="cri-o://58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6" gracePeriod=30
Jul 24 18:29:23.006319 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.006293    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:23.111633 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.111356    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:23.189449 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.189429    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"
Jul 24 18:29:23.242350 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.242329    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"
Jul 24 18:29:23.278690 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.278669    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7llsj\" (UniqueName: \"kubernetes.io/projected/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc-kube-api-access-7llsj\") pod \"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc\" (UID: \"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc\") "
Jul 24 18:29:23.278812 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.278755    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc-plugins-config-volume\") pod \"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc\" (UID: \"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc\") "
Jul 24 18:29:23.279206 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.279172    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68f4a2ef-d9ea-4c0c-9395-a7673bef32fc" (UID: "68f4a2ef-d9ea-4c0c-9395-a7673bef32fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:29:23.280600 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.280562    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc-kube-api-access-7llsj" (OuterVolumeSpecName: "kube-api-access-7llsj") pod "68f4a2ef-d9ea-4c0c-9395-a7673bef32fc" (UID: "68f4a2ef-d9ea-4c0c-9395-a7673bef32fc"). InnerVolumeSpecName "kube-api-access-7llsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:29:23.379593 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.379544    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bppl2\" (UniqueName: \"kubernetes.io/projected/d69b6d93-5170-40ca-9920-6f5e33c2aa21-kube-api-access-bppl2\") pod \"d69b6d93-5170-40ca-9920-6f5e33c2aa21\" (UID: \"d69b6d93-5170-40ca-9920-6f5e33c2aa21\") "
Jul 24 18:29:23.379746 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.379671    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69b6d93-5170-40ca-9920-6f5e33c2aa21-plugins-config-volume\") pod \"d69b6d93-5170-40ca-9920-6f5e33c2aa21\" (UID: \"d69b6d93-5170-40ca-9920-6f5e33c2aa21\") "
Jul 24 18:29:23.379858 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.379847    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:29:23.379896 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.379860    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7llsj\" (UniqueName: \"kubernetes.io/projected/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc-kube-api-access-7llsj\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:29:23.380055 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.380022    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69b6d93-5170-40ca-9920-6f5e33c2aa21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d69b6d93-5170-40ca-9920-6f5e33c2aa21" (UID: "d69b6d93-5170-40ca-9920-6f5e33c2aa21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:29:23.381658 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.381641    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69b6d93-5170-40ca-9920-6f5e33c2aa21-kube-api-access-bppl2" (OuterVolumeSpecName: "kube-api-access-bppl2") pod "d69b6d93-5170-40ca-9920-6f5e33c2aa21" (UID: "d69b6d93-5170-40ca-9920-6f5e33c2aa21"). InnerVolumeSpecName "kube-api-access-bppl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:29:23.481084 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.481007    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bppl2\" (UniqueName: \"kubernetes.io/projected/d69b6d93-5170-40ca-9920-6f5e33c2aa21-kube-api-access-bppl2\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:29:23.481084 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:23.481036    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69b6d93-5170-40ca-9920-6f5e33c2aa21-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:29:24.026405 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.026366    2581 generic.go:356] "Generic (PLEG): container finished" podID="68f4a2ef-d9ea-4c0c-9395-a7673bef32fc" containerID="82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47" exitCode=0
Jul 24 18:29:24.026874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.026428    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"
Jul 24 18:29:24.026874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.026453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf" event={"ID":"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc","Type":"ContainerDied","Data":"82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47"}
Jul 24 18:29:24.026874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.026516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf" event={"ID":"68f4a2ef-d9ea-4c0c-9395-a7673bef32fc","Type":"ContainerDied","Data":"43fbcd52f23de75f8741c6cf741bef46004ece334c6af901b7ac8b93e72ce4bf"}
Jul 24 18:29:24.026874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.026545    2581 scope.go:117] "RemoveContainer" containerID="82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47"
Jul 24 18:29:24.027908 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.027804    2581 generic.go:356] "Generic (PLEG): container finished" podID="d69b6d93-5170-40ca-9920-6f5e33c2aa21" containerID="58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6" exitCode=0
Jul 24 18:29:24.027908 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.027856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7" event={"ID":"d69b6d93-5170-40ca-9920-6f5e33c2aa21","Type":"ContainerDied","Data":"58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6"}
Jul 24 18:29:24.028036 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.027913    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7" event={"ID":"d69b6d93-5170-40ca-9920-6f5e33c2aa21","Type":"ContainerDied","Data":"e11517b24976b0815d8a90f944a9926f19a6b0a26b601c13f6cfd67ef35f2305"}
Jul 24 18:29:24.028036 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.027869    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"
Jul 24 18:29:24.035510 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.035395    2581 scope.go:117] "RemoveContainer" containerID="82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47"
Jul 24 18:29:24.035739 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:29:24.035714    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47\": container with ID starting with 82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47 not found: ID does not exist" containerID="82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47"
Jul 24 18:29:24.035809 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.035769    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47"} err="failed to get container status \"82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47\": rpc error: code = NotFound desc = could not find container \"82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47\": container with ID starting with 82035e0c6468af0c2cbf35c030388f01345d04627111b53d12835c1f41497c47 not found: ID does not exist"
Jul 24 18:29:24.035809 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.035790    2581 scope.go:117] "RemoveContainer" containerID="58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6"
Jul 24 18:29:24.043459 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.043440    2581 scope.go:117] "RemoveContainer" containerID="58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6"
Jul 24 18:29:24.043688 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:29:24.043667    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6\": container with ID starting with 58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6 not found: ID does not exist" containerID="58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6"
Jul 24 18:29:24.043746 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.043696    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6"} err="failed to get container status \"58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6\": rpc error: code = NotFound desc = could not find container \"58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6\": container with ID starting with 58b7ae28405c493991f2956ae0687f0535140bd3c21e03f15ba31d660f2468f6 not found: ID does not exist"
Jul 24 18:29:24.051420 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.051399    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"]
Jul 24 18:29:24.054206 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.054187    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7ee2e6-8cdd89848-kwcc7"]
Jul 24 18:29:24.064931 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.064912    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"]
Jul 24 18:29:24.069153 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.069134    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7ee2e6-6dd4dd6f98-g6tlf"]
Jul 24 18:29:24.959150 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.959117    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68f4a2ef-d9ea-4c0c-9395-a7673bef32fc" path="/var/lib/kubelet/pods/68f4a2ef-d9ea-4c0c-9395-a7673bef32fc/volumes"
Jul 24 18:29:24.959477 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:24.959464    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d69b6d93-5170-40ca-9920-6f5e33c2aa21" path="/var/lib/kubelet/pods/d69b6d93-5170-40ca-9920-6f5e33c2aa21/volumes"
Jul 24 18:29:27.494400 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:27.494366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:27.852744 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:27.852710    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:29:32.941366 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:29:32.941327    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:09.326270 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:09.326235    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:09.972629 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:09.972565    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:15.613589 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:15.613535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:21.494994 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:21.494923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:21.572494 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:21.572461    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:21.790072 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:21.789988    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:22.004341 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:22.004311    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:25.844886 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:25.844857    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:32.726189 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:32.726155    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"]
Jul 24 18:30:32.726627 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:32.726406    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2" podUID="4ee62993-dcdb-4fee-a3b3-afeb1a3ab364" containerName="maas-api" containerID="cri-o://633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c" gracePeriod=30
Jul 24 18:30:32.979397 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:32.979345    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:30:33.067086 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.067055    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-maas-api-tls\") pod \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") "
Jul 24 18:30:33.067245 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.067114    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-openshift-service-ca\") pod \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") "
Jul 24 18:30:33.067245 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.067158    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l22pd\" (UniqueName: \"kubernetes.io/projected/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-kube-api-access-l22pd\") pod \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\" (UID: \"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364\") "
Jul 24 18:30:33.067478 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.067454    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ee62993-dcdb-4fee-a3b3-afeb1a3ab364" (UID: "4ee62993-dcdb-4fee-a3b3-afeb1a3ab364"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:30:33.069147 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.069124    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ee62993-dcdb-4fee-a3b3-afeb1a3ab364" (UID: "4ee62993-dcdb-4fee-a3b3-afeb1a3ab364"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:30:33.069272 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.069252    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-kube-api-access-l22pd" (OuterVolumeSpecName: "kube-api-access-l22pd") pod "4ee62993-dcdb-4fee-a3b3-afeb1a3ab364" (UID: "4ee62993-dcdb-4fee-a3b3-afeb1a3ab364"). InnerVolumeSpecName "kube-api-access-l22pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:30:33.168063 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.168034    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:30:33.168063 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.168058    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:30:33.168226 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.168069    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l22pd\" (UniqueName: \"kubernetes.io/projected/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364-kube-api-access-l22pd\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:30:33.345902 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.345869    2581 generic.go:356] "Generic (PLEG): container finished" podID="4ee62993-dcdb-4fee-a3b3-afeb1a3ab364" containerID="633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c" exitCode=0
Jul 24 18:30:33.346059 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.345913    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"
Jul 24 18:30:33.346059 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.345955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2" event={"ID":"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364","Type":"ContainerDied","Data":"633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c"}
Jul 24 18:30:33.346059 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.346011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2" event={"ID":"4ee62993-dcdb-4fee-a3b3-afeb1a3ab364","Type":"ContainerDied","Data":"93b799f33e9682b2e78f6a3cbc434e34a8d9ffea982dee54e1b608ff721bb690"}
Jul 24 18:30:33.346059 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.346033    2581 scope.go:117] "RemoveContainer" containerID="633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c"
Jul 24 18:30:33.355097 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.355079    2581 scope.go:117] "RemoveContainer" containerID="633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c"
Jul 24 18:30:33.355329 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:30:33.355311    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c\": container with ID starting with 633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c not found: ID does not exist" containerID="633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c"
Jul 24 18:30:33.355434 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.355335    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c"} err="failed to get container status \"633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c\": rpc error: code = NotFound desc = could not find container \"633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c\": container with ID starting with 633a9b1db4a066e15e09e9e51a53d72c12362ea6e4b61bafed5bd2b0e4078a3c not found: ID does not exist"
Jul 24 18:30:33.373542 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.373519    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"]
Jul 24 18:30:33.379561 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.379540    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e18e67-69cd5598-g6ls2"]
Jul 24 18:30:33.461242 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:33.461211    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:34.846726 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:34.846694    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:34.957756 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:34.957729    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ee62993-dcdb-4fee-a3b3-afeb1a3ab364" path="/var/lib/kubelet/pods/4ee62993-dcdb-4fee-a3b3-afeb1a3ab364/volumes"
Jul 24 18:30:37.200563 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:37.200522    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:37.581693 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:37.581662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:43.252464 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:43.252427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:43.353044 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:43.353009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:43.547623 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:43.547538    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:43.756992 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:43.756954    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:54.356558 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.356519    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"]
Jul 24 18:30:54.357046 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.356823    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6" podUID="7b86cebe-90d2-4940-b323-eddcc6c06c5a" containerName="maas-api" containerID="cri-o://0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25" gracePeriod=30
Jul 24 18:30:54.385978 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.385951    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:54.416818 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.416792    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"]
Jul 24 18:30:54.417010 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.416986    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw" podUID="e24f0986-e94e-4538-b1ff-d9df35e8445f" containerName="payload-processing" containerID="cri-o://991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278" gracePeriod=30
Jul 24 18:30:54.423995 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.423971    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"]
Jul 24 18:30:54.424213 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.424168    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb" podUID="f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f" containerName="payload-pre-processing" containerID="cri-o://f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5" gracePeriod=30
Jul 24 18:30:54.486262 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.486233    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:54.596073 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.595396    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:54.700383 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.700355    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:30:54.723255 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.723030    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"
Jul 24 18:30:54.726321 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.726292    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"
Jul 24 18:30:54.734265 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.734241    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw4dn\" (UniqueName: \"kubernetes.io/projected/7b86cebe-90d2-4940-b323-eddcc6c06c5a-kube-api-access-qw4dn\") pod \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") "
Jul 24 18:30:54.734387 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.734300    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b86cebe-90d2-4940-b323-eddcc6c06c5a-maas-api-tls\") pod \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") "
Jul 24 18:30:54.734387 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.734331    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86cebe-90d2-4940-b323-eddcc6c06c5a-openshift-service-ca\") pod \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\" (UID: \"7b86cebe-90d2-4940-b323-eddcc6c06c5a\") "
Jul 24 18:30:54.734387 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.734355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f-plugins-config-volume\") pod \"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f\" (UID: \"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f\") "
Jul 24 18:30:54.734552 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.734424    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crbx5\" (UniqueName: \"kubernetes.io/projected/e24f0986-e94e-4538-b1ff-d9df35e8445f-kube-api-access-crbx5\") pod \"e24f0986-e94e-4538-b1ff-d9df35e8445f\" (UID: \"e24f0986-e94e-4538-b1ff-d9df35e8445f\") "
Jul 24 18:30:54.734552 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.734453    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24f0986-e94e-4538-b1ff-d9df35e8445f-plugins-config-volume\") pod \"e24f0986-e94e-4538-b1ff-d9df35e8445f\" (UID: \"e24f0986-e94e-4538-b1ff-d9df35e8445f\") "
Jul 24 18:30:54.734552 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.734495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkkzv\" (UniqueName: \"kubernetes.io/projected/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f-kube-api-access-xkkzv\") pod \"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f\" (UID: \"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f\") "
Jul 24 18:30:54.734739 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.734707    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b86cebe-90d2-4940-b323-eddcc6c06c5a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b86cebe-90d2-4940-b323-eddcc6c06c5a" (UID: "7b86cebe-90d2-4940-b323-eddcc6c06c5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:30:54.734802 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.734744    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f" (UID: "f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:30:54.734855 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.734801    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86cebe-90d2-4940-b323-eddcc6c06c5a-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:30:54.735210 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.735185    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e24f0986-e94e-4538-b1ff-d9df35e8445f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e24f0986-e94e-4538-b1ff-d9df35e8445f" (UID: "e24f0986-e94e-4538-b1ff-d9df35e8445f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:30:54.736542 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.736520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b86cebe-90d2-4940-b323-eddcc6c06c5a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b86cebe-90d2-4940-b323-eddcc6c06c5a" (UID: "7b86cebe-90d2-4940-b323-eddcc6c06c5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:30:54.736703 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.736681    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24f0986-e94e-4538-b1ff-d9df35e8445f-kube-api-access-crbx5" (OuterVolumeSpecName: "kube-api-access-crbx5") pod "e24f0986-e94e-4538-b1ff-d9df35e8445f" (UID: "e24f0986-e94e-4538-b1ff-d9df35e8445f"). InnerVolumeSpecName "kube-api-access-crbx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:30:54.736930 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.736904    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f-kube-api-access-xkkzv" (OuterVolumeSpecName: "kube-api-access-xkkzv") pod "f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f" (UID: "f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f"). InnerVolumeSpecName "kube-api-access-xkkzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:30:54.737013 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.736994    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b86cebe-90d2-4940-b323-eddcc6c06c5a-kube-api-access-qw4dn" (OuterVolumeSpecName: "kube-api-access-qw4dn") pod "7b86cebe-90d2-4940-b323-eddcc6c06c5a" (UID: "7b86cebe-90d2-4940-b323-eddcc6c06c5a"). InnerVolumeSpecName "kube-api-access-qw4dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:30:54.835746 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.835718    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:30:54.835746 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.835747    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crbx5\" (UniqueName: \"kubernetes.io/projected/e24f0986-e94e-4538-b1ff-d9df35e8445f-kube-api-access-crbx5\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:30:54.835980 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.835760    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24f0986-e94e-4538-b1ff-d9df35e8445f-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:30:54.835980 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.835773    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkkzv\" (UniqueName: \"kubernetes.io/projected/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f-kube-api-access-xkkzv\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:30:54.835980 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.835786    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw4dn\" (UniqueName: \"kubernetes.io/projected/7b86cebe-90d2-4940-b323-eddcc6c06c5a-kube-api-access-qw4dn\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:30:54.835980 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:54.835798    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b86cebe-90d2-4940-b323-eddcc6c06c5a-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:30:55.446464 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.446431    2581 generic.go:356] "Generic (PLEG): container finished" podID="e24f0986-e94e-4538-b1ff-d9df35e8445f" containerID="991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278" exitCode=0
Jul 24 18:30:55.446893 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.446474    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"
Jul 24 18:30:55.446893 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.446509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw" event={"ID":"e24f0986-e94e-4538-b1ff-d9df35e8445f","Type":"ContainerDied","Data":"991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278"}
Jul 24 18:30:55.446893 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.446587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw" event={"ID":"e24f0986-e94e-4538-b1ff-d9df35e8445f","Type":"ContainerDied","Data":"d076caaa1048b2e84ecc7f7ae3275854ab1100793feed13cfb3e04c28d6e285d"}
Jul 24 18:30:55.446893 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.446625    2581 scope.go:117] "RemoveContainer" containerID="991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278"
Jul 24 18:30:55.448098 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.447888    2581 generic.go:356] "Generic (PLEG): container finished" podID="7b86cebe-90d2-4940-b323-eddcc6c06c5a" containerID="0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25" exitCode=0
Jul 24 18:30:55.448098 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.447942    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"
Jul 24 18:30:55.448098 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.447973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6" event={"ID":"7b86cebe-90d2-4940-b323-eddcc6c06c5a","Type":"ContainerDied","Data":"0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25"}
Jul 24 18:30:55.448098 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.448013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6" event={"ID":"7b86cebe-90d2-4940-b323-eddcc6c06c5a","Type":"ContainerDied","Data":"e3d16e3960e91d516e9ba15b50b9a1cdba4d211307201c43700e1a9da5b44083"}
Jul 24 18:30:55.449760 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.449737    2581 generic.go:356] "Generic (PLEG): container finished" podID="f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f" containerID="f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5" exitCode=0
Jul 24 18:30:55.449900 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.449784    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"
Jul 24 18:30:55.449900 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.449826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb" event={"ID":"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f","Type":"ContainerDied","Data":"f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5"}
Jul 24 18:30:55.449900 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.449869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb" event={"ID":"f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f","Type":"ContainerDied","Data":"fc86fa781c407d1508fc36845bc75bfb46a57040d0df810cfab6be60621c5e38"}
Jul 24 18:30:55.456116 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.456097    2581 scope.go:117] "RemoveContainer" containerID="991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278"
Jul 24 18:30:55.456379 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:30:55.456357    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278\": container with ID starting with 991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278 not found: ID does not exist" containerID="991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278"
Jul 24 18:30:55.456432 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.456389    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278"} err="failed to get container status \"991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278\": rpc error: code = NotFound desc = could not find container \"991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278\": container with ID starting with 991dde6cb069f3835ee3faacd11648ba1186556623082878d17b675506e07278 not found: ID does not exist"
Jul 24 18:30:55.456432 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.456407    2581 scope.go:117] "RemoveContainer" containerID="0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25"
Jul 24 18:30:55.463669 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.463653    2581 scope.go:117] "RemoveContainer" containerID="0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25"
Jul 24 18:30:55.463910 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:30:55.463892    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25\": container with ID starting with 0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25 not found: ID does not exist" containerID="0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25"
Jul 24 18:30:55.463961 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.463916    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25"} err="failed to get container status \"0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25\": rpc error: code = NotFound desc = could not find container \"0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25\": container with ID starting with 0adef7a88aab988d0960164ebd464ba2265d962726545b86ca51b803015e2a25 not found: ID does not exist"
Jul 24 18:30:55.463961 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.463933    2581 scope.go:117] "RemoveContainer" containerID="f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5"
Jul 24 18:30:55.468661 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.468643    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"]
Jul 24 18:30:55.471278 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.471251    2581 scope.go:117] "RemoveContainer" containerID="f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5"
Jul 24 18:30:55.471520 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:30:55.471492    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5\": container with ID starting with f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5 not found: ID does not exist" containerID="f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5"
Jul 24 18:30:55.471631 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.471521    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5"} err="failed to get container status \"f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5\": rpc error: code = NotFound desc = could not find container \"f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5\": container with ID starting with f6429e86e9042dbe268ebe509a3e439dd713f30f1c67b62d9b3ebc93067ac6b5 not found: ID does not exist"
Jul 24 18:30:55.473825 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.473805    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-100cff-65b98b4798-s7csw"]
Jul 24 18:30:55.486363 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.486339    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"]
Jul 24 18:30:55.490669 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.490648    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-100cff-58cd65d5b-48pdb"]
Jul 24 18:30:55.501505 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.501485    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"]
Jul 24 18:30:55.506136 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:55.506118    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-100cff-6f6f8b5495-p4bg6"]
Jul 24 18:30:56.962734 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:56.962705    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b86cebe-90d2-4940-b323-eddcc6c06c5a" path="/var/lib/kubelet/pods/7b86cebe-90d2-4940-b323-eddcc6c06c5a/volumes"
Jul 24 18:30:56.963102 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:56.963042    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e24f0986-e94e-4538-b1ff-d9df35e8445f" path="/var/lib/kubelet/pods/e24f0986-e94e-4538-b1ff-d9df35e8445f/volumes"
Jul 24 18:30:56.963444 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:56.963426    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f" path="/var/lib/kubelet/pods/f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f/volumes"
Jul 24 18:30:58.976106 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:58.976071    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:30:59.359857 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:30:59.359825    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:05.640283 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:05.640251    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:05.741334 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:05.741296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:05.913109 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:05.913025    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"]
Jul 24 18:31:05.913273 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:05.913226    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb" podUID="233b69e0-5dd4-4235-a7f4-3bff6c32f89c" containerName="istio-proxy" containerID="cri-o://27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440" gracePeriod=30
Jul 24 18:31:05.957439 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:05.957399    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:06.159638 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:06.159604    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:11.167867 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.167843    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:31:11.263654 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.263589    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-token\") pod \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") "
Jul 24 18:31:11.263654 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.263635    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-envoy\") pod \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") "
Jul 24 18:31:11.263831 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.263664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-workload-socket\") pod \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") "
Jul 24 18:31:11.263831 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.263708    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rwcs\" (UniqueName: \"kubernetes.io/projected/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-kube-api-access-2rwcs\") pod \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") "
Jul 24 18:31:11.263831 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.263770    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-workload-certs\") pod \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") "
Jul 24 18:31:11.263831 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.263795    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istiod-ca-cert\") pod \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") "
Jul 24 18:31:11.263831 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.263828    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-credential-socket\") pod \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") "
Jul 24 18:31:11.264094 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.263881    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-podinfo\") pod \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") "
Jul 24 18:31:11.264094 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.263905    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-data\") pod \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\" (UID: \"233b69e0-5dd4-4235-a7f4-3bff6c32f89c\") "
Jul 24 18:31:11.264094 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.263945    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "233b69e0-5dd4-4235-a7f4-3bff6c32f89c" (UID: "233b69e0-5dd4-4235-a7f4-3bff6c32f89c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:31:11.264094 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.264036    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "233b69e0-5dd4-4235-a7f4-3bff6c32f89c" (UID: "233b69e0-5dd4-4235-a7f4-3bff6c32f89c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:31:11.264307 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.264133    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "233b69e0-5dd4-4235-a7f4-3bff6c32f89c" (UID: "233b69e0-5dd4-4235-a7f4-3bff6c32f89c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:31:11.264307 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.264152    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "233b69e0-5dd4-4235-a7f4-3bff6c32f89c" (UID: "233b69e0-5dd4-4235-a7f4-3bff6c32f89c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:31:11.264307 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.264212    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:11.264307 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.264233    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:11.264970 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.264947    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-data" (OuterVolumeSpecName: "istio-data") pod "233b69e0-5dd4-4235-a7f4-3bff6c32f89c" (UID: "233b69e0-5dd4-4235-a7f4-3bff6c32f89c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:31:11.265786 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.265755    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "233b69e0-5dd4-4235-a7f4-3bff6c32f89c" (UID: "233b69e0-5dd4-4235-a7f4-3bff6c32f89c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:31:11.265880 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.265818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-token" (OuterVolumeSpecName: "istio-token") pod "233b69e0-5dd4-4235-a7f4-3bff6c32f89c" (UID: "233b69e0-5dd4-4235-a7f4-3bff6c32f89c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:31:11.266346 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.266321    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "233b69e0-5dd4-4235-a7f4-3bff6c32f89c" (UID: "233b69e0-5dd4-4235-a7f4-3bff6c32f89c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 18:31:11.266346 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.266325    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-kube-api-access-2rwcs" (OuterVolumeSpecName: "kube-api-access-2rwcs") pod "233b69e0-5dd4-4235-a7f4-3bff6c32f89c" (UID: "233b69e0-5dd4-4235-a7f4-3bff6c32f89c"). InnerVolumeSpecName "kube-api-access-2rwcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:31:11.365115 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.365089    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:11.365115 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.365111    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:11.365263 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.365120    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:11.365263 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.365130    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:11.365263 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.365138    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:11.365263 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.365145    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:11.365263 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.365153    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rwcs\" (UniqueName: \"kubernetes.io/projected/233b69e0-5dd4-4235-a7f4-3bff6c32f89c-kube-api-access-2rwcs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:11.525553 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.525470    2581 generic.go:356] "Generic (PLEG): container finished" podID="233b69e0-5dd4-4235-a7f4-3bff6c32f89c" containerID="27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440" exitCode=0
Jul 24 18:31:11.525553 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.525524    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"
Jul 24 18:31:11.525746 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.525556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb" event={"ID":"233b69e0-5dd4-4235-a7f4-3bff6c32f89c","Type":"ContainerDied","Data":"27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440"}
Jul 24 18:31:11.525746 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.525646    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb" event={"ID":"233b69e0-5dd4-4235-a7f4-3bff6c32f89c","Type":"ContainerDied","Data":"bf8b87f006a3803505d90e0dfed69ab3094e0f526e03fd45c4d9a5ff1e87d20e"}
Jul 24 18:31:11.525746 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.525669    2581 scope.go:117] "RemoveContainer" containerID="27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440"
Jul 24 18:31:11.534268 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.534250    2581 scope.go:117] "RemoveContainer" containerID="942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5"
Jul 24 18:31:11.541452 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.541432    2581 scope.go:117] "RemoveContainer" containerID="27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440"
Jul 24 18:31:11.541774 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:31:11.541755    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440\": container with ID starting with 27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440 not found: ID does not exist" containerID="27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440"
Jul 24 18:31:11.541858 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.541786    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440"} err="failed to get container status \"27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440\": rpc error: code = NotFound desc = could not find container \"27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440\": container with ID starting with 27eab6011aaaebf3ec69a45bb658d9bdd7bab42b3ca82636f153fff3a940a440 not found: ID does not exist"
Jul 24 18:31:11.541858 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.541808    2581 scope.go:117] "RemoveContainer" containerID="942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5"
Jul 24 18:31:11.542059 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:31:11.542042    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5\": container with ID starting with 942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5 not found: ID does not exist" containerID="942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5"
Jul 24 18:31:11.542098 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.542065    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5"} err="failed to get container status \"942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5\": rpc error: code = NotFound desc = could not find container \"942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5\": container with ID starting with 942ca8acf574e84ec0da1c0386d127c1d804b4e9cb677d57e4d4310f3a0d7fd5 not found: ID does not exist"
Jul 24 18:31:11.546602 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.546584    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"]
Jul 24 18:31:11.552324 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:11.552302    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-100cff-openshift-default-7869d9748-cqmgb"]
Jul 24 18:31:12.958136 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:12.958092    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="233b69e0-5dd4-4235-a7f4-3bff6c32f89c" path="/var/lib/kubelet/pods/233b69e0-5dd4-4235-a7f4-3bff6c32f89c/volumes"
Jul 24 18:31:16.832812 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:16.832780    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"]
Jul 24 18:31:16.833228 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:16.832979    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j" podUID="602b34f9-782d-4eb8-9a10-51e824605d0a" containerName="maas-api" containerID="cri-o://f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb" gracePeriod=30
Jul 24 18:31:16.860208 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:16.860184    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:16.894847 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:16.894821    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"]
Jul 24 18:31:16.895029 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:16.895005    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg" podUID="ceb1d686-0057-4b5e-85a1-8672e91c6e54" containerName="payload-pre-processing" containerID="cri-o://c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8" gracePeriod=30
Jul 24 18:31:16.961305 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:16.961279    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:17.059786 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.059717    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:17.149433 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.149407    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:31:17.185212 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.185189    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"
Jul 24 18:31:17.209803 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.209776    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ng8h\" (UniqueName: \"kubernetes.io/projected/602b34f9-782d-4eb8-9a10-51e824605d0a-kube-api-access-6ng8h\") pod \"602b34f9-782d-4eb8-9a10-51e824605d0a\" (UID: \"602b34f9-782d-4eb8-9a10-51e824605d0a\") "
Jul 24 18:31:17.209920 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.209857    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602b34f9-782d-4eb8-9a10-51e824605d0a-maas-api-tls\") pod \"602b34f9-782d-4eb8-9a10-51e824605d0a\" (UID: \"602b34f9-782d-4eb8-9a10-51e824605d0a\") "
Jul 24 18:31:17.209920 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.209908    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602b34f9-782d-4eb8-9a10-51e824605d0a-openshift-service-ca\") pod \"602b34f9-782d-4eb8-9a10-51e824605d0a\" (UID: \"602b34f9-782d-4eb8-9a10-51e824605d0a\") "
Jul 24 18:31:17.210245 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.210213    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602b34f9-782d-4eb8-9a10-51e824605d0a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "602b34f9-782d-4eb8-9a10-51e824605d0a" (UID: "602b34f9-782d-4eb8-9a10-51e824605d0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:31:17.211748 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.211722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602b34f9-782d-4eb8-9a10-51e824605d0a-kube-api-access-6ng8h" (OuterVolumeSpecName: "kube-api-access-6ng8h") pod "602b34f9-782d-4eb8-9a10-51e824605d0a" (UID: "602b34f9-782d-4eb8-9a10-51e824605d0a"). InnerVolumeSpecName "kube-api-access-6ng8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:31:17.211748 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.211741    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602b34f9-782d-4eb8-9a10-51e824605d0a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "602b34f9-782d-4eb8-9a10-51e824605d0a" (UID: "602b34f9-782d-4eb8-9a10-51e824605d0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:31:17.310653 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.310622    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psx9n\" (UniqueName: \"kubernetes.io/projected/ceb1d686-0057-4b5e-85a1-8672e91c6e54-kube-api-access-psx9n\") pod \"ceb1d686-0057-4b5e-85a1-8672e91c6e54\" (UID: \"ceb1d686-0057-4b5e-85a1-8672e91c6e54\") "
Jul 24 18:31:17.310828 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.310664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb1d686-0057-4b5e-85a1-8672e91c6e54-plugins-config-volume\") pod \"ceb1d686-0057-4b5e-85a1-8672e91c6e54\" (UID: \"ceb1d686-0057-4b5e-85a1-8672e91c6e54\") "
Jul 24 18:31:17.310902 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.310871    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602b34f9-782d-4eb8-9a10-51e824605d0a-maas-api-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:17.310902 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.310892    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602b34f9-782d-4eb8-9a10-51e824605d0a-openshift-service-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:17.311024 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.310908    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ng8h\" (UniqueName: \"kubernetes.io/projected/602b34f9-782d-4eb8-9a10-51e824605d0a-kube-api-access-6ng8h\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:17.311135 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.311109    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb1d686-0057-4b5e-85a1-8672e91c6e54-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ceb1d686-0057-4b5e-85a1-8672e91c6e54" (UID: "ceb1d686-0057-4b5e-85a1-8672e91c6e54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:31:17.312654 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.312632    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb1d686-0057-4b5e-85a1-8672e91c6e54-kube-api-access-psx9n" (OuterVolumeSpecName: "kube-api-access-psx9n") pod "ceb1d686-0057-4b5e-85a1-8672e91c6e54" (UID: "ceb1d686-0057-4b5e-85a1-8672e91c6e54"). InnerVolumeSpecName "kube-api-access-psx9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:31:17.412120 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.412047    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psx9n\" (UniqueName: \"kubernetes.io/projected/ceb1d686-0057-4b5e-85a1-8672e91c6e54-kube-api-access-psx9n\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:17.412120 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.412079    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb1d686-0057-4b5e-85a1-8672e91c6e54-plugins-config-volume\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:17.553641 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.553608    2581 generic.go:356] "Generic (PLEG): container finished" podID="602b34f9-782d-4eb8-9a10-51e824605d0a" containerID="f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb" exitCode=0
Jul 24 18:31:17.553782 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.553665    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"
Jul 24 18:31:17.553782 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.553681    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j" event={"ID":"602b34f9-782d-4eb8-9a10-51e824605d0a","Type":"ContainerDied","Data":"f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb"}
Jul 24 18:31:17.553782 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.553746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j" event={"ID":"602b34f9-782d-4eb8-9a10-51e824605d0a","Type":"ContainerDied","Data":"65d6d32d57335eca02f84e47553a8483b19eb923e41b944a7eba68de97aa1954"}
Jul 24 18:31:17.553782 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.553774    2581 scope.go:117] "RemoveContainer" containerID="f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb"
Jul 24 18:31:17.554969 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.554946    2581 generic.go:356] "Generic (PLEG): container finished" podID="ceb1d686-0057-4b5e-85a1-8672e91c6e54" containerID="c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8" exitCode=0
Jul 24 18:31:17.555074 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.554983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg" event={"ID":"ceb1d686-0057-4b5e-85a1-8672e91c6e54","Type":"ContainerDied","Data":"c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8"}
Jul 24 18:31:17.555074 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.555005    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"
Jul 24 18:31:17.555074 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.555041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg" event={"ID":"ceb1d686-0057-4b5e-85a1-8672e91c6e54","Type":"ContainerDied","Data":"bb6162c7c14605e37a93df48c626b78bde46d315266c15c813c9ed7bd795f3a3"}
Jul 24 18:31:17.562777 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.562760    2581 scope.go:117] "RemoveContainer" containerID="f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb"
Jul 24 18:31:17.563084 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:31:17.563061    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb\": container with ID starting with f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb not found: ID does not exist" containerID="f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb"
Jul 24 18:31:17.563182 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.563093    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb"} err="failed to get container status \"f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb\": rpc error: code = NotFound desc = could not find container \"f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb\": container with ID starting with f6d8c60f7ab0745f00afe1e8438c3b089f26416f23c43e93a3423d1ddd7755fb not found: ID does not exist"
Jul 24 18:31:17.563182 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.563117    2581 scope.go:117] "RemoveContainer" containerID="c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8"
Jul 24 18:31:17.570406 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.570389    2581 scope.go:117] "RemoveContainer" containerID="c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8"
Jul 24 18:31:17.570643 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:31:17.570624    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8\": container with ID starting with c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8 not found: ID does not exist" containerID="c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8"
Jul 24 18:31:17.570696 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.570650    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8"} err="failed to get container status \"c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8\": rpc error: code = NotFound desc = could not find container \"c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8\": container with ID starting with c4e3e9868233b3a139a1f2500df30e1d8d22a7612aede46f4d482aac0a6898a8 not found: ID does not exist"
Jul 24 18:31:17.580354 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.580333    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"]
Jul 24 18:31:17.583273 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.583253    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4d732-6b8977cbc5-r2n5j"]
Jul 24 18:31:17.593724 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.593699    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"]
Jul 24 18:31:17.597731 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:17.597710    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c4d732-55776cc5bd-9vtdg"]
Jul 24 18:31:18.957779 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:18.957744    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="602b34f9-782d-4eb8-9a10-51e824605d0a" path="/var/lib/kubelet/pods/602b34f9-782d-4eb8-9a10-51e824605d0a/volumes"
Jul 24 18:31:18.958142 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:18.958084    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb1d686-0057-4b5e-85a1-8672e91c6e54" path="/var/lib/kubelet/pods/ceb1d686-0057-4b5e-85a1-8672e91c6e54/volumes"
Jul 24 18:31:21.359961 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:21.359902    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:21.761857 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:21.761773    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:27.878671 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:27.878636    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:27.944819 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:27.944783    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:28.167275 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:28.167188    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"]
Jul 24 18:31:28.167466 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:28.167439    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j" podUID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerName="istio-proxy" containerID="cri-o://c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f" gracePeriod=30
Jul 24 18:31:28.206137 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:28.206112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:28.377681 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:28.377647    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:33.417799 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.417775    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:31:33.547176 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547098    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg24t\" (UniqueName: \"kubernetes.io/projected/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-kube-api-access-dg24t\") pod \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") "
Jul 24 18:31:33.547176 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547130    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-data\") pod \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") "
Jul 24 18:31:33.547176 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547160    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-credential-socket\") pod \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") "
Jul 24 18:31:33.547176 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547178    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-workload-certs\") pod \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") "
Jul 24 18:31:33.547511 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547205    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istiod-ca-cert\") pod \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") "
Jul 24 18:31:33.547511 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-workload-socket\") pod \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") "
Jul 24 18:31:33.547511 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547272    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-token\") pod \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") "
Jul 24 18:31:33.547511 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-envoy\") pod \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") "
Jul 24 18:31:33.547511 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547340    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-podinfo\") pod \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\" (UID: \"2ce832ed-08d8-4587-88a8-c6e9ce5af9be\") "
Jul 24 18:31:33.547511 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547492    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ce832ed-08d8-4587-88a8-c6e9ce5af9be" (UID: "2ce832ed-08d8-4587-88a8-c6e9ce5af9be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:31:33.547874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547540    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ce832ed-08d8-4587-88a8-c6e9ce5af9be" (UID: "2ce832ed-08d8-4587-88a8-c6e9ce5af9be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:31:33.547874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547561    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ce832ed-08d8-4587-88a8-c6e9ce5af9be" (UID: "2ce832ed-08d8-4587-88a8-c6e9ce5af9be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:31:33.547874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547621    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:33.547874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.547664    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ce832ed-08d8-4587-88a8-c6e9ce5af9be" (UID: "2ce832ed-08d8-4587-88a8-c6e9ce5af9be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:31:33.548372 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.548346    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ce832ed-08d8-4587-88a8-c6e9ce5af9be" (UID: "2ce832ed-08d8-4587-88a8-c6e9ce5af9be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:31:33.549351 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.549318    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-kube-api-access-dg24t" (OuterVolumeSpecName: "kube-api-access-dg24t") pod "2ce832ed-08d8-4587-88a8-c6e9ce5af9be" (UID: "2ce832ed-08d8-4587-88a8-c6e9ce5af9be"). InnerVolumeSpecName "kube-api-access-dg24t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:31:33.549455 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.549364    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ce832ed-08d8-4587-88a8-c6e9ce5af9be" (UID: "2ce832ed-08d8-4587-88a8-c6e9ce5af9be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:31:33.549514 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.549500    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ce832ed-08d8-4587-88a8-c6e9ce5af9be" (UID: "2ce832ed-08d8-4587-88a8-c6e9ce5af9be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:31:33.549588 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.549559    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ce832ed-08d8-4587-88a8-c6e9ce5af9be" (UID: "2ce832ed-08d8-4587-88a8-c6e9ce5af9be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 18:31:33.625195 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.625163    2581 generic.go:356] "Generic (PLEG): container finished" podID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerID="c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f" exitCode=0
Jul 24 18:31:33.625297 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.625222    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"
Jul 24 18:31:33.625297 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.625259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j" event={"ID":"2ce832ed-08d8-4587-88a8-c6e9ce5af9be","Type":"ContainerDied","Data":"c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f"}
Jul 24 18:31:33.625436 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.625331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j" event={"ID":"2ce832ed-08d8-4587-88a8-c6e9ce5af9be","Type":"ContainerDied","Data":"00ace0896ddb8cd6d1b5eb9110de1975d4779b6c39da20f1fcf990024965be99"}
Jul 24 18:31:33.625436 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.625375    2581 scope.go:117] "RemoveContainer" containerID="c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f"
Jul 24 18:31:33.633492 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.633478    2581 scope.go:117] "RemoveContainer" containerID="4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166"
Jul 24 18:31:33.640412 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.640395    2581 scope.go:117] "RemoveContainer" containerID="c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f"
Jul 24 18:31:33.640655 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:31:33.640636    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f\": container with ID starting with c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f not found: ID does not exist" containerID="c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f"
Jul 24 18:31:33.640732 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.640667    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f"} err="failed to get container status \"c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f\": rpc error: code = NotFound desc = could not find container \"c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f\": container with ID starting with c082799546f946fb22c33fb28ebc6a59a9b432ff283325e29d7600c9b021331f not found: ID does not exist"
Jul 24 18:31:33.640732 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.640691    2581 scope.go:117] "RemoveContainer" containerID="4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166"
Jul 24 18:31:33.640942 ip-10-0-141-44 kubenswrapper[2581]: E0724 18:31:33.640922    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166\": container with ID starting with 4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166 not found: ID does not exist" containerID="4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166"
Jul 24 18:31:33.640990 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.640948    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166"} err="failed to get container status \"4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166\": rpc error: code = NotFound desc = could not find container \"4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166\": container with ID starting with 4ac837ae0cd472de11efa05b7874942d275fcf3762667e43f15c6bce0246d166 not found: ID does not exist"
Jul 24 18:31:33.646733 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.646713    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"]
Jul 24 18:31:33.648337 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.648321    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg24t\" (UniqueName: \"kubernetes.io/projected/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-kube-api-access-dg24t\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:33.648387 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.648341    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:33.648387 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.648350    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:33.648387 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.648360    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:33.648387 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.648368    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:33.648387 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.648377    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:33.648387 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.648386    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:33.648633 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.648394    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce832ed-08d8-4587-88a8-c6e9ce5af9be-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 24 18:31:33.650666 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:33.650649    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-c4d732-openshift-default-7b89687bb8-wxj4j"]
Jul 24 18:31:34.532989 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:34.532963    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d4bb4965f-bhxsx_b8bfbfd3-e3ad-4e7c-a922-f0e97d337188/manager/0.log"
Jul 24 18:31:34.903717 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:34.903693    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b794f588d-qc7qd_c3260159-f143-4de8-83fe-75c5fe49a976/copy-manifests/0.log"
Jul 24 18:31:34.923067 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:34.923042    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b794f588d-qc7qd_c3260159-f143-4de8-83fe-75c5fe49a976/manager/0.log"
Jul 24 18:31:34.957515 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:34.957488    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" path="/var/lib/kubelet/pods/2ce832ed-08d8-4587-88a8-c6e9ce5af9be/volumes"
Jul 24 18:31:35.787159 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:35.787124    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748615-ldj8z_c099e9d6-dc3a-4750-8d02-2e106b7c48bc/cleanup/0.log"
Jul 24 18:31:38.593561 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:38.593533    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kzv5g_081313fb-605f-4b48-beaa-202a611482dc/discovery/0.log"
Jul 24 18:31:39.688652 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:39.688618    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-g59ht_a1b93aab-418a-45fe-9015-8200a69da666/main/0.log"
Jul 24 18:31:39.923113 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:39.923081    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-hnl8t_a776e008-7fad-4d1a-ab9d-913b48fff221/main/0.log"
Jul 24 18:31:45.615870 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:45.615831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:45.712703 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:45.712667    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log" limit=1024
Jul 24 18:31:48.610821 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:48.610791    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zdzrg_e79a4ac8-d313-44f1-a654-ef419b6a1109/global-pull-secret-syncer/0.log"
Jul 24 18:31:48.673411 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:48.673380    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8p2dj_e72c8a23-8926-4a30-a022-178ed7196e5d/konnectivity-agent/0.log"
Jul 24 18:31:48.836812 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:48.836784    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-44.ec2.internal_fc5bec91994039b1fb9327533b8ad0c7/haproxy/0.log"
Jul 24 18:31:57.348832 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:57.348801    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbr8m_7cba1bb4-6b1e-4e24-8738-1fafbb602fef/node-exporter/0.log"
Jul 24 18:31:57.373843 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:57.373823    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbr8m_7cba1bb4-6b1e-4e24-8738-1fafbb602fef/kube-rbac-proxy/0.log"
Jul 24 18:31:57.397143 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:57.397126    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbr8m_7cba1bb4-6b1e-4e24-8738-1fafbb602fef/init-textfile/0.log"
Jul 24 18:31:58.787361 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787332    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"]
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787612    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f4a2ef-d9ea-4c0c-9395-a7673bef32fc" containerName="payload-processing"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787624    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f4a2ef-d9ea-4c0c-9395-a7673bef32fc" containerName="payload-processing"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787633    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e76a999-eba4-4b65-8251-afc4e6821cac" containerName="payload-processing"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787638    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e76a999-eba4-4b65-8251-afc4e6821cac" containerName="payload-processing"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787649    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d69b6d93-5170-40ca-9920-6f5e33c2aa21" containerName="payload-pre-processing"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787656    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d69b6d93-5170-40ca-9920-6f5e33c2aa21" containerName="payload-pre-processing"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787665    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233b69e0-5dd4-4235-a7f4-3bff6c32f89c" containerName="istio-proxy"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787670    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="233b69e0-5dd4-4235-a7f4-3bff6c32f89c" containerName="istio-proxy"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787678    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e24f0986-e94e-4538-b1ff-d9df35e8445f" containerName="payload-processing"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787683    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e24f0986-e94e-4538-b1ff-d9df35e8445f" containerName="payload-processing"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787691    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb1d686-0057-4b5e-85a1-8672e91c6e54" containerName="payload-pre-processing"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787696    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb1d686-0057-4b5e-85a1-8672e91c6e54" containerName="payload-pre-processing"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787703    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="602b34f9-782d-4eb8-9a10-51e824605d0a" containerName="maas-api"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787707    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="602b34f9-782d-4eb8-9a10-51e824605d0a" containerName="maas-api"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787715    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee62993-dcdb-4fee-a3b3-afeb1a3ab364" containerName="maas-api"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787720    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee62993-dcdb-4fee-a3b3-afeb1a3ab364" containerName="maas-api"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787728    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118dbac4-656e-4bdb-954f-dbea4fdc6c78" containerName="maas-api"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787732    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="118dbac4-656e-4bdb-954f-dbea4fdc6c78" containerName="maas-api"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787744    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f" containerName="payload-pre-processing"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787749    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f" containerName="payload-pre-processing"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787755    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b86cebe-90d2-4940-b323-eddcc6c06c5a" containerName="maas-api"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787760    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b86cebe-90d2-4940-b323-eddcc6c06c5a" containerName="maas-api"
Jul 24 18:31:58.787754 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787766    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerName="istio-proxy"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787770    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerName="istio-proxy"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787776    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerName="istio-proxy"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787781    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerName="istio-proxy"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787823    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb1d686-0057-4b5e-85a1-8672e91c6e54" containerName="payload-pre-processing"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787831    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b86cebe-90d2-4940-b323-eddcc6c06c5a" containerName="maas-api"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787837    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f4a2ef-d9ea-4c0c-9395-a7673bef32fc" containerName="payload-processing"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787842    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d69b6d93-5170-40ca-9920-6f5e33c2aa21" containerName="payload-pre-processing"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787847    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="233b69e0-5dd4-4235-a7f4-3bff6c32f89c" containerName="istio-proxy"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787853    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="233b69e0-5dd4-4235-a7f4-3bff6c32f89c" containerName="istio-proxy"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787859    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="602b34f9-782d-4eb8-9a10-51e824605d0a" containerName="maas-api"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787865    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerName="istio-proxy"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787870    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="118dbac4-656e-4bdb-954f-dbea4fdc6c78" containerName="maas-api"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787875    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ee62993-dcdb-4fee-a3b3-afeb1a3ab364" containerName="maas-api"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787879    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ba7042-c1f7-44cf-9604-dbb2ae0ae94f" containerName="payload-pre-processing"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787883    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e24f0986-e94e-4538-b1ff-d9df35e8445f" containerName="payload-processing"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787888    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e76a999-eba4-4b65-8251-afc4e6821cac" containerName="payload-processing"
Jul 24 18:31:58.788463 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.787893    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ce832ed-08d8-4587-88a8-c6e9ce5af9be" containerName="istio-proxy"
Jul 24 18:31:58.790832 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.790811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.793184 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.793160    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dwr8f\"/\"kube-root-ca.crt\""
Jul 24 18:31:58.793448 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.793432    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dwr8f\"/\"openshift-service-ca.crt\""
Jul 24 18:31:58.794117 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.794102    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dwr8f\"/\"default-dockercfg-85vn6\""
Jul 24 18:31:58.800691 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.800672    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"]
Jul 24 18:31:58.842346 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.842317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6303fdd-5b7e-4be1-ab59-80663def6ea5-sys\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.842346 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.842342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d6303fdd-5b7e-4be1-ab59-80663def6ea5-podres\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.842538 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.842372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4bkq\" (UniqueName: \"kubernetes.io/projected/d6303fdd-5b7e-4be1-ab59-80663def6ea5-kube-api-access-b4bkq\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.842538 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.842434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d6303fdd-5b7e-4be1-ab59-80663def6ea5-proc\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.842538 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.842490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6303fdd-5b7e-4be1-ab59-80663def6ea5-lib-modules\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.943674 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.943642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6303fdd-5b7e-4be1-ab59-80663def6ea5-sys\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.943674 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.943675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d6303fdd-5b7e-4be1-ab59-80663def6ea5-podres\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.943874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.943706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4bkq\" (UniqueName: \"kubernetes.io/projected/d6303fdd-5b7e-4be1-ab59-80663def6ea5-kube-api-access-b4bkq\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.943874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.943760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6303fdd-5b7e-4be1-ab59-80663def6ea5-sys\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.943874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.943796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d6303fdd-5b7e-4be1-ab59-80663def6ea5-proc\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.943874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.943827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6303fdd-5b7e-4be1-ab59-80663def6ea5-lib-modules\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.943874 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.943826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d6303fdd-5b7e-4be1-ab59-80663def6ea5-podres\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.944096 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.943883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d6303fdd-5b7e-4be1-ab59-80663def6ea5-proc\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.944096 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.943960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6303fdd-5b7e-4be1-ab59-80663def6ea5-lib-modules\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:58.951916 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:58.951897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4bkq\" (UniqueName: \"kubernetes.io/projected/d6303fdd-5b7e-4be1-ab59-80663def6ea5-kube-api-access-b4bkq\") pod \"perf-node-gather-daemonset-8gqhc\" (UID: \"d6303fdd-5b7e-4be1-ab59-80663def6ea5\") " pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:59.100355 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:59.100330    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:59.225241 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:59.225216    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"]
Jul 24 18:31:59.227705 ip-10-0-141-44 kubenswrapper[2581]: W0724 18:31:59.227679    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd6303fdd_5b7e_4be1_ab59_80663def6ea5.slice/crio-952da6e933cfbc2a95deef6ff34c1952f4622860be2e77e3bdc9122772b85e48 WatchSource:0}: Error finding container 952da6e933cfbc2a95deef6ff34c1952f4622860be2e77e3bdc9122772b85e48: Status 404 returned error can't find the container with id 952da6e933cfbc2a95deef6ff34c1952f4622860be2e77e3bdc9122772b85e48
Jul 24 18:31:59.346123 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:59.346103    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-v8577_d497c07a-1396-47b5-9c37-0d91fc12c7cf/networking-console-plugin/0.log"
Jul 24 18:31:59.734617 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:59.734584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc" event={"ID":"d6303fdd-5b7e-4be1-ab59-80663def6ea5","Type":"ContainerStarted","Data":"b68aca35e669ff0c5d96674b80387875d5f631604430a0c2d20e264b39c4bb38"}
Jul 24 18:31:59.734807 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:59.734634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc" event={"ID":"d6303fdd-5b7e-4be1-ab59-80663def6ea5","Type":"ContainerStarted","Data":"952da6e933cfbc2a95deef6ff34c1952f4622860be2e77e3bdc9122772b85e48"}
Jul 24 18:31:59.734807 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:59.734682    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:31:59.751508 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:31:59.751454    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc" podStartSLOduration=1.751437267 podStartE2EDuration="1.751437267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:31:59.749680563 +0000 UTC m=+3333.266191599" watchObservedRunningTime="2026-07-24 18:31:59.751437267 +0000 UTC m=+3333.267948303"
Jul 24 18:32:02.367414 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:02.367383    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cxc52_01447c5b-a7fc-43e5-b0f0-a0c43ee232d9/dns/0.log"
Jul 24 18:32:02.390532 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:02.390502    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cxc52_01447c5b-a7fc-43e5-b0f0-a0c43ee232d9/kube-rbac-proxy/0.log"
Jul 24 18:32:02.580357 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:02.580320    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r9gdx_c47e251f-f0c5-48d9-80bd-b09ad0bcea70/dns-node-resolver/0.log"
Jul 24 18:32:03.208761 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:03.208733    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gnvrc_3724c750-78f9-43b2-bf42-eb3d0d7a6c4b/node-ca/0.log"
Jul 24 18:32:04.351818 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:04.351789    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kzv5g_081313fb-605f-4b48-beaa-202a611482dc/discovery/0.log"
Jul 24 18:32:05.406821 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:05.406794    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2tb94_c3f39de7-d644-4bd1-b0c2-0f1d8aab01f7/serve-healthcheck-canary/0.log"
Jul 24 18:32:05.747914 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:05.747844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dwr8f/perf-node-gather-daemonset-8gqhc"
Jul 24 18:32:06.003580 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:06.003490    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fchlk_d812cd21-578e-4bb6-884c-c452e3b2c174/kube-rbac-proxy/0.log"
Jul 24 18:32:06.027187 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:06.027164    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fchlk_d812cd21-578e-4bb6-884c-c452e3b2c174/exporter/0.log"
Jul 24 18:32:06.050581 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:06.050547    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fchlk_d812cd21-578e-4bb6-884c-c452e3b2c174/extractor/0.log"
Jul 24 18:32:09.208957 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:09.208926    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d4bb4965f-bhxsx_b8bfbfd3-e3ad-4e7c-a922-f0e97d337188/manager/0.log"
Jul 24 18:32:09.390209 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:09.390179    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b794f588d-qc7qd_c3260159-f143-4de8-83fe-75c5fe49a976/manager/0.log"
Jul 24 18:32:09.419938 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:09.419908    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b794f588d-qc7qd_c3260159-f143-4de8-83fe-75c5fe49a976/copy-manifests/0.log"
Jul 24 18:32:10.372672 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:10.372645    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kzv5g_081313fb-605f-4b48-beaa-202a611482dc/discovery/0.log"
Jul 24 18:32:11.064428 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:11.064399    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b4bb5cddb-m8qqp_ce937f90-f289-4673-8768-90dd0d57ac25/manager/0.log"
Jul 24 18:32:11.122253 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:11.122222    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-qhs8j_64581efe-8049-44d3-baa4-e31fb0241b14/openshift-lws-operator/0.log"
Jul 24 18:32:12.319921 ip-10-0-141-44 kubenswrapper[2581]: I0724 18:32:12.319881    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kzv5g_081313fb-605f-4b48-beaa-202a611482dc/discovery/0.log"