Jul 31 01:37:21.521831 ip-10-0-141-142 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 01:37:21.895139 ip-10-0-141-142 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 01:37:21.895139 ip-10-0-141-142 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 01:37:21.895139 ip-10-0-141-142 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 01:37:21.895139 ip-10-0-141-142 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 01:37:21.895139 ip-10-0-141-142 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 01:37:21.895907 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.895781    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 01:37:21.902267 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902242    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 01:37:21.902267 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902263    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 01:37:21.902267 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902268    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 01:37:21.902267 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902271    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 01:37:21.902267 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902274    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902277    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902282    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902284    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902287    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902291    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902294    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902297    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902299    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902302    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902305    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902308    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902311    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902314    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902316    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902319    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902321    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902324    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902326    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902329    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 01:37:21.902463 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902332    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902334    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902337    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902339    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902342    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902345    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902347    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902352    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902355    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902357    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902360    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902364    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902368    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902371    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902374    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902377    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902379    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902382    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902385    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902387    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 01:37:21.902968 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902389    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902392    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902394    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902397    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902399    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902402    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902406    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902410    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902413    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902416    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902419    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902421    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902424    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902427    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902429    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902432    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902434    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902437    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902440    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 01:37:21.903466 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902442    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902444    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902447    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902450    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902452    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902455    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902457    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902460    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902463    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902466    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902468    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902471    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902474    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902477    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902479    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902482    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902484    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902487    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902490    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902492    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 01:37:21.903935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902500    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902504    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902507    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902509    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902512    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902514    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902518    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902521    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902524    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902527    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902530    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902533    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902536    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902538    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902541    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.902543    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903318    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903325    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903328    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 01:37:21.904432 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903331    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903335    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903338    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903341    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903343    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903346    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903348    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903351    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903353    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903356    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903359    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903361    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903364    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903366    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903374    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903377    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903379    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903382    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903385    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903387    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 01:37:21.904942 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903389    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903392    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903394    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903397    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903399    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903402    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903404    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903406    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903410    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903412    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903415    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903417    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903420    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903423    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903425    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903427    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903431    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903435    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903438    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 01:37:21.905437 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903440    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903443    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903445    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903448    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903450    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903453    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903455    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903458    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903468    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903472    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903475    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903478    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903481    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903484    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903486    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903489    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903492    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903494    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903497    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 01:37:21.905929 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903499    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903502    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903504    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903507    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903510    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903513    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903517    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903520    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903523    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903525    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903527    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903530    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903533    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903535    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903538    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903540    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903542    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903545    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903547    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903550    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 01:37:21.906395 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903552    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903577    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903587    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903600    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903603    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903606    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903608    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903610    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903613    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903615    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903618    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903621    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903623    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903626    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903628    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903631    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903634    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.903637    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904061    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904066    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 01:37:21.906914 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904069    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904072    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904074    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904077    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904080    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904082    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904085    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904087    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904090    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904092    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904095    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904097    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904100    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904102    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904110    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904113    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904115    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904118    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904120    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 01:37:21.907422 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904123    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904125    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904127    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904131    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904133    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904136    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904138    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904141    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904144    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904146    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904149    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904151    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904154    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904156    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904158    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904161    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904163    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904166    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904169    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904171    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 01:37:21.907921 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904173    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904176    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904178    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904181    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904183    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904186    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904188    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904192    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904201    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904204    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904207    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904210    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904212    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904215    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904218    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904220    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904225    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904228    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904230    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 01:37:21.908399 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904233    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904236    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904239    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904242    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904244    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904247    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904249    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904252    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904255    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904259    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904263    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904266    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904269    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904274    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904281    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904284    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904287    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904289    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904292    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 01:37:21.908909 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904294    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904297    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904299    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904306    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904309    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904311    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904314    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904316    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904318    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904321    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904324    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904326    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904329    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904332    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904334    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904337    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904339    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904342    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904344    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.904346    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 01:37:21.909418 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904420    2556 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904427    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904440    2556 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904457    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904461    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904464    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904468    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904474    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904479    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904482    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904486    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904489    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904492    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904495    2556 flags.go:64] FLAG: --cgroup-root=""
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904498    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904500    2556 flags.go:64] FLAG: --client-ca-file=""
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904509    2556 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904512    2556 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904518    2556 flags.go:64] FLAG: --cluster-domain=""
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904521    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904524    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904527    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904530    2556 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904537    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 01:37:21.909933 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904540    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904543    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904546    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904549    2556 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904552    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904568    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904572    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904575    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904579    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904582    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904584    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904588    2556 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904591    2556 flags.go:64] FLAG: --enable-server="true"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904594    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904601    2556 flags.go:64] FLAG: --event-burst="100"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904604    2556 flags.go:64] FLAG: --event-qps="50"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904607    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904610    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904613    2556 flags.go:64] FLAG: --eviction-hard=""
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904617    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904619    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904622    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904625    2556 flags.go:64] FLAG: --eviction-soft=""
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904628    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904631    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 01:37:21.910504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904639    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904642    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904645    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904648    2556 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904650    2556 flags.go:64] FLAG: --feature-gates=""
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904654    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904658    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904661    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904665    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904668    2556 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904671    2556 flags.go:64] FLAG: --help="false"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904673    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-141-142.ec2.internal"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904676    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904679    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904682    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904685    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904689    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904691    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904696    2556 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904699    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904701    2556 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904704    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904707    2556 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904710    2556 flags.go:64] FLAG: --kube-reserved=""
Jul 31 01:37:21.911180 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904713    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904715    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904718    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904721    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904723    2556 flags.go:64] FLAG: --lock-file=""
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904726    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904729    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904732    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904736    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904744    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904747    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904750    2556 flags.go:64] FLAG: --logging-format="text"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904753    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904756    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904761    2556 flags.go:64] FLAG: --manifest-url=""
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904763    2556 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904767    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904770    2556 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904774    2556 flags.go:64] FLAG: --max-pods="110"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904777    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904780    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904783    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904786    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904788    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904791    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 01:37:21.911816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904794    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904803    2556 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904806    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904809    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904812    2556 flags.go:64] FLAG: --pod-cidr=""
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904814    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904820    2556 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904823    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904826    2556 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904829    2556 flags.go:64] FLAG: --port="10250"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904832    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904835    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04cce552195feb240"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904838    2556 flags.go:64] FLAG: --qos-reserved=""
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904840    2556 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904843    2556 flags.go:64] FLAG: --register-node="true"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904846    2556 flags.go:64] FLAG: --register-with-taints=""
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904849    2556 flags.go:64] FLAG: --registry-burst="10"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904859    2556 flags.go:64] FLAG: --registry-qps="5"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904862    2556 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904864    2556 flags.go:64] FLAG: --reserved-memory=""
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904868    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904871    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904875    2556 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904878    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904881    2556 flags.go:64] FLAG: --runonce="false"
Jul 31 01:37:21.912427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904883    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904886    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904889    2556 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904892    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904894    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904898    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904901    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904904    2556 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904908    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904911    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904913    2556 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904916    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904919    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904922    2556 flags.go:64] FLAG: --system-cgroups=""
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904924    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904930    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904932    2556 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904935    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904941    2556 flags.go:64] FLAG: --tls-min-version=""
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904944    2556 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904947    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904949    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904953    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904955    2556 flags.go:64] FLAG: --v="2"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904959    2556 flags.go:64] FLAG: --version="false"
Jul 31 01:37:21.913063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904968    2556 flags.go:64] FLAG: --vmodule=""
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904972    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.904975    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905099    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905104    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905107    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905110    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905113    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905116    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905118    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905120    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905123    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905125    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905128    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905131    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905134    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905136    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905139    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905141    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 01:37:21.913686 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905144    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905146    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905149    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905152    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905154    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905156    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905159    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905161    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905164    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905166    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905169    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905171    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905174    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905176    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905184    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905187    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905189    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905195    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905197    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905200    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 01:37:21.914188 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905202    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905205    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905207    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905210    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905212    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905216    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905219    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905222    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905226    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905229    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905231    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905234    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905236    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905239    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905242    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905244    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905246    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905249    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905251    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 01:37:21.914782 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905254    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905256    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905259    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905262    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905264    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905266    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905269    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905271    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905274    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905277    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905281    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905284    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905287    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905291    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905294    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905297    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905299    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905302    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905304    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905307    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 01:37:21.915249 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905309    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905313    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905315    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905318    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905320    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905322    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905325    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905328    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905330    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905333    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905336    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905339    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905342    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905345    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905348    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905351    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905353    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905355    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905358    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 01:37:21.915755 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905360    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905363    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905366    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905369    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.905372    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.905943    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.913112    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.913127    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.913147    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.913153    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913226    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913232    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913235    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913238    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913241    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 01:37:21.916210 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913244    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913247    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913250    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913253    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913256    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913259    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913262    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913265    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913268    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913270    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913273    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913275    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913279    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913283    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913286    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913290    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913292    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913295    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913298    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 01:37:21.916663 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913302    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913304    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913307    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913309    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913312    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913315    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913318    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913320    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913323    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913326    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913328    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913331    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913333    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913336    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913338    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913341    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913344    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913346    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913348    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913351    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 01:37:21.917127 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913353    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913356    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913358    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913361    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913363    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913365    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913368    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913370    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913373    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913375    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913378    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913380    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913383    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913386    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913388    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913390    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913393    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913397    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913399    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 01:37:21.917629 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913402    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913404    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913407    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913409    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913412    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913416    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913419    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913422    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913425    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913428    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913431    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913433    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913436    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913439    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913441    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913444    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913447    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913451    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913453    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913456    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 01:37:21.918115 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913458    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913460    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913463    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913465    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913468    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913470    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913473    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913475    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913478    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913480    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913484    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913486    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913489    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913491    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913494    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 01:37:21.918691 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913496    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.913501    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913669    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913676    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913679    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913681    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913684    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913687    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913690    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913692    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913695    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913699    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913702    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913704    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 01:37:21.919053 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913707    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913710    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913712    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913714    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913717    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913719    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913722    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913724    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913726    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913729    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913732    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913740    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913743    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913751    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913754    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913756    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913759    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913762    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913764    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913767    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 01:37:21.919408 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913769    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913772    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913774    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913777    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913779    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913782    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913784    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913788    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913792    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913796    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913800    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913803    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913805    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913808    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913811    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913814    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913816    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913819    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 01:37:21.919924 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913821    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913824    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913826    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913829    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913831    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913834    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913843    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913847    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913850    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913852    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913854    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913857    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913859    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913862    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913864    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913867    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913869    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913872    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913874    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913876    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 01:37:21.920382 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913879    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913881    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913883    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913886    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913888    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913891    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913893    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913896    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913898    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913900    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913903    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913905    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913908    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913910    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913912    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913915    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913918    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913920    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913922    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913931    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 01:37:21.920888 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913933    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913936    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913938    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913941    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913943    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913945    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913948    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913950    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:21.913953    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.913957    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.914068    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.916169    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.917502    2556 server.go:1019] "Starting client certificate rotation"
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.917641    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 01:37:21.921384 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.917678    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 01:37:21.921779 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.920086    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 01:37:21.922197 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.922184    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 01:37:21.933116 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.933096    2556 log.go:25] "Validated CRI v1 runtime API"
Jul 31 01:37:21.935166 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.935151    2556 log.go:25] "Validated CRI v1 image API"
Jul 31 01:37:21.936530 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.936515    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 01:37:21.939535 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.939513    2556 fs.go:135] Filesystem UUIDs: map[1849f773-5b3b-4b9b-b02e-e502371c75a0:/dev/nvme0n1p3 27ff6785-53a6-4a19-b291-8a44e3da7077:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 31 01:37:21.939632 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.939534    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 01:37:21.944934 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.944815    2556 manager.go:216] Machine: {Timestamp:2026-07-31 01:37:21.943625088 +0000 UTC m=+0.309370708 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100291 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e746df9831f59f83155b5b24e7315 SystemUUID:ec2e746d-f983-1f59-f831-55b5b24e7315 BootID:979df613-627e-4b00-849d-a328f9b56293 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9a:86:da:9f:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9a:86:da:9f:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:45:13:3c:73:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 01:37:21.944934 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.944927    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 01:37:21.945096 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.945035    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 01:37:21.946636 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.946620    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 01:37:21.947662 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.947636    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 01:37:21.947799 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.947667    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 01:37:21.948371 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.948362    2556 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 01:37:21.948419 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.948375    2556 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 01:37:21.948419 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.948383    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 01:37:21.948948 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.948936    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 01:37:21.949680 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.949670    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 01:37:21.953752 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.953741    2556 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 01:37:21.953875 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.953866    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 01:37:21.953910 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.953899    2556 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 01:37:21.953941 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.953911    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 01:37:21.953941 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.953933    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 01:37:21.953992 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.953946    2556 kubelet.go:398] "Adding apiserver pod source"
Jul 31 01:37:21.953992 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.953958    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 01:37:21.954373 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.954362    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 01:37:21.954411 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.954381    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 01:37:21.956296 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.956273    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 01:37:21.957756 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.957743    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 01:37:21.957809 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.957773    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 01:37:21.958915 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.958904    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 01:37:21.958948 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.958919    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 01:37:21.958948 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.958925    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 01:37:21.958948 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.958930    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 01:37:21.958948 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.958938    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 01:37:21.958948 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.958945    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 01:37:21.959077 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.958956    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 01:37:21.959077 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.958964    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 01:37:21.959077 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.958985    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 01:37:21.959077 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.958994    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 01:37:21.959077 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.959006    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 01:37:21.959077 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.959014    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 01:37:21.959826 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.959814    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 01:37:21.959859 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.959827    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 01:37:21.962943 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.962928    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 01:37:21.962943 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:21.962936    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 01:37:21.963043 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:21.962935    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 01:37:21.963289 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.963268    2556 server.go:1268] "Started kubelet"
Jul 31 01:37:21.963393 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.963317    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 01:37:21.963574 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.963530    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 01:37:21.963711 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.963690    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 01:37:21.964128 ip-10-0-141-142 systemd[1]: Started Kubernetes Kubelet.
Jul 31 01:37:21.964541 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.964514    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k8fpt"
Jul 31 01:37:21.964937 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.964920    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 01:37:21.970687 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.970215    2556 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 01:37:21.970779 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:21.970661    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 01:37:21.972437 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.972420    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k8fpt"
Jul 31 01:37:21.973192 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.973170    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 01:37:21.973271 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.973196    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 01:37:21.973327 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:21.973272    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:21.974062 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.974045    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 01:37:21.974062 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.974047    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 01:37:21.974212 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.974073    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 01:37:21.974212 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.974154    2556 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 01:37:21.974212 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.974164    2556 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 01:37:21.974390 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.974377    2556 factory.go:153] Registering CRI-O factory
Jul 31 01:37:21.974476 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.974395    2556 factory.go:223] Registration of the crio container factory successfully
Jul 31 01:37:21.974476 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.974445    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 01:37:21.974476 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.974455    2556 factory.go:55] Registering systemd factory
Jul 31 01:37:21.974476 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.974462    2556 factory.go:223] Registration of the systemd container factory successfully
Jul 31 01:37:21.974746 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.974484    2556 factory.go:103] Registering Raw factory
Jul 31 01:37:21.974746 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.974497    2556 manager.go:1196] Started watching for new ooms in manager
Jul 31 01:37:21.974746 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:21.973739    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-142.ec2.internal.18c73c15e23b8e26  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-142.ec2.internal,UID:ip-10-0-141-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-142.ec2.internal,},FirstTimestamp:2026-07-31 01:37:21.963224614 +0000 UTC m=+0.328970233,LastTimestamp:2026-07-31 01:37:21.963224614 +0000 UTC m=+0.328970233,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-142.ec2.internal,}"
Jul 31 01:37:21.975131 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.975118    2556 manager.go:349] Starting recovery of all containers
Jul 31 01:37:21.977616 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:21.977593    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 01:37:21.977721 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:21.977628    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 01:37:21.986662 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.986644    2556 manager.go:354] Recovery completed
Jul 31 01:37:21.991193 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.991080    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 01:37:21.993403 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.993381    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 01:37:21.993475 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.993416    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 01:37:21.993475 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.993427    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Jul 31 01:37:21.994021 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.994008    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 01:37:21.994021 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.994018    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 01:37:21.994164 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.994031    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 01:37:21.994164 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.994033    2556 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 01:37:21.994712 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.994695    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 01:37:21.994761 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.994729    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 01:37:21.994761 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.994744    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Jul 31 01:37:21.995967 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.995954    2556 policy_none.go:49] "None policy: Start"
Jul 31 01:37:21.996030 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.995973    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 01:37:21.996030 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.995986    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 01:37:21.997840 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:21.997828    2556 policy_none.go:47] "Start"
Jul 31 01:37:22.035152 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.035128    2556 manager.go:337] "Starting Device Plugin manager"
Jul 31 01:37:22.035244 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.035164    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 01:37:22.035244 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.035176    2556 server.go:87] "Starting device plugin registration server"
Jul 31 01:37:22.035457 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.035445    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 01:37:22.035500 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.035458    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 01:37:22.035613 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.035590    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 01:37:22.035732 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.035681    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 01:37:22.035732 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.035689    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 01:37:22.036187 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.036162    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 01:37:22.036244 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.036225    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:22.136114 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.136062    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 01:37:22.138101 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.137113    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 01:37:22.138101 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.137147    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 01:37:22.138101 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.137158    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Jul 31 01:37:22.138101 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.137188    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.143446 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.143421    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 01:37:22.144916 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.144899    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 01:37:22.144997 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.144940    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 01:37:22.144997 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.144961    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 01:37:22.144997 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.144971    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.145127 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.144995    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 01:37:22.145127 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.144997    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-142.ec2.internal\": node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:22.147023 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.146966    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 01:37:22.188511 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.188482    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:22.245708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.245672    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"]
Jul 31 01:37:22.245824 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.245752    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 01:37:22.249427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.249407    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 01:37:22.249539 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.249439    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 01:37:22.249539 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.249452    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Jul 31 01:37:22.251672 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.251659    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 01:37:22.251793 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.251771    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.251852 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.251813    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 01:37:22.252504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.252488    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 01:37:22.252581 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.252516    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 01:37:22.252581 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.252527    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 01:37:22.252581 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.252550    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 01:37:22.252723 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.252586    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Jul 31 01:37:22.252723 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.252531    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Jul 31 01:37:22.254791 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.254775    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.254848 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.254804    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 01:37:22.255491 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.255476    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 01:37:22.255588 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.255499    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 01:37:22.255588 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.255509    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Jul 31 01:37:22.275946 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.275922    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f923c99785cedeeb9d5a1e0081b46a58-config\") pod \"kube-apiserver-proxy-ip-10-0-141-142.ec2.internal\" (UID: \"f923c99785cedeeb9d5a1e0081b46a58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.276052 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.275954    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a631120ad11f3f853e9f18ea275f194-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal\" (UID: \"6a631120ad11f3f853e9f18ea275f194\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.276052 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.275981    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a631120ad11f3f853e9f18ea275f194-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal\" (UID: \"6a631120ad11f3f853e9f18ea275f194\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.280286 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.280268    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-142.ec2.internal\" not found" node="ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.285076 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.285059    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-142.ec2.internal\" not found" node="ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.289099 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.289085    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:22.376996 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.376960    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f923c99785cedeeb9d5a1e0081b46a58-config\") pod \"kube-apiserver-proxy-ip-10-0-141-142.ec2.internal\" (UID: \"f923c99785cedeeb9d5a1e0081b46a58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.376996 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.376990    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a631120ad11f3f853e9f18ea275f194-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal\" (UID: \"6a631120ad11f3f853e9f18ea275f194\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.376996 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.377006    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a631120ad11f3f853e9f18ea275f194-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal\" (UID: \"6a631120ad11f3f853e9f18ea275f194\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.377254 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.377045    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a631120ad11f3f853e9f18ea275f194-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal\" (UID: \"6a631120ad11f3f853e9f18ea275f194\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.377254 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.377074    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a631120ad11f3f853e9f18ea275f194-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal\" (UID: \"6a631120ad11f3f853e9f18ea275f194\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.377254 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.377075    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f923c99785cedeeb9d5a1e0081b46a58-config\") pod \"kube-apiserver-proxy-ip-10-0-141-142.ec2.internal\" (UID: \"f923c99785cedeeb9d5a1e0081b46a58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.390069 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.390046    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:22.491069 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.490991    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:22.582531 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.582504    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.587712 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.587553    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:22.591136 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.591045    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:22.691551 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.691514    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:22.792149 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.792081    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:22.892671 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.892640    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:22.916821 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.916789    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 01:37:22.917257 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.916934    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 01:37:22.977204 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.977163    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 01:32:22 +0000 UTC" deadline="2028-03-27 14:54:47.340034874 +0000 UTC"
Jul 31 01:37:22.977204 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:22.977194    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14533h17m24.362843506s"
Jul 31 01:37:22.993185 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:22.993166    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:23.076957 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.076929    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 01:37:23.093754 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:23.093717    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Jul 31 01:37:23.118520 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:23.118484    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf923c99785cedeeb9d5a1e0081b46a58.slice/crio-bbc1d09c9def3b89c39279b4cc56203c7440fbee605ddd9db242e77b5ff58a9d WatchSource:0}: Error finding container bbc1d09c9def3b89c39279b4cc56203c7440fbee605ddd9db242e77b5ff58a9d: Status 404 returned error can't find the container with id bbc1d09c9def3b89c39279b4cc56203c7440fbee605ddd9db242e77b5ff58a9d
Jul 31 01:37:23.118791 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:23.118771    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a631120ad11f3f853e9f18ea275f194.slice/crio-5caa71f9e848f53aa703d2821926e174e6e8613a5b0be127338d7630e2e0268f WatchSource:0}: Error finding container 5caa71f9e848f53aa703d2821926e174e6e8613a5b0be127338d7630e2e0268f: Status 404 returned error can't find the container with id 5caa71f9e848f53aa703d2821926e174e6e8613a5b0be127338d7630e2e0268f
Jul 31 01:37:23.123383 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.123369    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 01:37:23.137521 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.137501    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 01:37:23.148387 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.148342    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal" event={"ID":"6a631120ad11f3f853e9f18ea275f194","Type":"ContainerStarted","Data":"5caa71f9e848f53aa703d2821926e174e6e8613a5b0be127338d7630e2e0268f"}
Jul 31 01:37:23.149339 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.149317    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal" event={"ID":"f923c99785cedeeb9d5a1e0081b46a58","Type":"ContainerStarted","Data":"bbc1d09c9def3b89c39279b4cc56203c7440fbee605ddd9db242e77b5ff58a9d"}
Jul 31 01:37:23.175495 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.175451    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:23.183853 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.183831    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 01:37:23.184817 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.184804    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Jul 31 01:37:23.196746 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.196724    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 01:37:23.413505 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.413413    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 01:37:23.955366 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.955330    2556 apiserver.go:52] "Watching apiserver"
Jul 31 01:37:23.964584 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.964549    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 01:37:23.965754 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.965724    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-tqppw","openshift-image-registry/node-ca-lprjl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal","openshift-multus/multus-additional-cni-plugins-tq27z","kube-system/global-pull-secret-syncer-dnk4v","kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj","openshift-multus/multus-5l5wj","openshift-multus/network-metrics-daemon-x94zh","openshift-network-diagnostics/network-check-target-vv7wq","openshift-network-operator/iptables-alerter-685q7","openshift-ovn-kubernetes/ovnkube-node-fsprs","kube-system/konnectivity-agent-88pgt","openshift-cluster-node-tuning-operator/tuned-mvmpt"]
Jul 31 01:37:23.969052 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.968624    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:23.971209 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.970922    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-88pgt"
Jul 31 01:37:23.971209 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.970942    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 01:37:23.971209 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.971153    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 01:37:23.971209 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.971191    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5bj4r\""
Jul 31 01:37:23.971461 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.971311    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 01:37:23.973008 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.972960    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 01:37:23.973116 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.973040    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qvj9l\""
Jul 31 01:37:23.973116 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.973050    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 01:37:23.973696 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.973673    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.973855 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.973835    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 01:37:23.975783 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.975761    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-965cs\""
Jul 31 01:37:23.975887 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.975764    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 01:37:23.975959 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.975940    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 01:37:23.977657 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.977207    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:23.979586 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.979569    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mjqm5\""
Jul 31 01:37:23.979709 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.979564    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:23.979709 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:23.979660    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:23.979883 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.979866    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 01:37:23.979951 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.979900    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 01:37:23.980193 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.980172    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 01:37:23.980306 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.980234    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 01:37:23.980433 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.980381    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 01:37:23.982653 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.982631    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 01:37:23.983125 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.983107    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:23.985085 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985045    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 01:37:23.985085 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985045    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jmrmw\""
Jul 31 01:37:23.985651 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985580    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-sys-fs\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:23.985743 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985655    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-run\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.985743 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985667    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:23.985743 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985683    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32bf4714-1aae-4a26-a0d3-cb99be970774-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:23.985743 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985711    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32bf4714-1aae-4a26-a0d3-cb99be970774-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:23.985743 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985737    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b46eb842-87fe-4021-b184-0ff856e849ad-tmp\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.986009 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:23.985740    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:23.986009 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985762    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjdg\" (UniqueName: \"kubernetes.io/projected/b46eb842-87fe-4021-b184-0ff856e849ad-kube-api-access-nvjdg\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.986009 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985797    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32bf4714-1aae-4a26-a0d3-cb99be970774-cnibin\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:23.986009 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985880    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:23.986009 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985933    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e26e8e31-84dc-4178-9f3d-172319dd06cb-konnectivity-ca\") pod \"konnectivity-agent-88pgt\" (UID: \"e26e8e31-84dc-4178-9f3d-172319dd06cb\") " pod="kube-system/konnectivity-agent-88pgt"
Jul 31 01:37:23.986009 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985954    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-sysctl-conf\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.986009 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985970    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-lib-modules\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.986009 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.985986    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b46eb842-87fe-4021-b184-0ff856e849ad-etc-tuned\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.986009 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986001    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32bf4714-1aae-4a26-a0d3-cb99be970774-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986017    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjwvr\" (UniqueName: \"kubernetes.io/projected/03ea1fd1-9975-468b-8297-41c0e36a9d00-kube-api-access-xjwvr\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986034    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e26e8e31-84dc-4178-9f3d-172319dd06cb-agent-certs\") pod \"konnectivity-agent-88pgt\" (UID: \"e26e8e31-84dc-4178-9f3d-172319dd06cb\") " pod="kube-system/konnectivity-agent-88pgt"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986071    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-var-lib-kubelet\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986096    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-modprobe-d\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986114    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32bf4714-1aae-4a26-a0d3-cb99be970774-os-release\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986129    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32bf4714-1aae-4a26-a0d3-cb99be970774-cni-binary-copy\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986149    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-registration-dir\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986164    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-device-dir\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986181    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45djd\" (UniqueName: \"kubernetes.io/projected/32bf4714-1aae-4a26-a0d3-cb99be970774-kube-api-access-45djd\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986197    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986217    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-sysconfig\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986233    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-kubernetes\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986248    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-systemd\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986261    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-sys\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986275    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-host\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.986459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986290    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-sysctl-d\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:23.987563 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986317    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32bf4714-1aae-4a26-a0d3-cb99be970774-system-cni-dir\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:23.987563 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.986332    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-socket-dir\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:23.988572 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.988551    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tqppw"
Jul 31 01:37:23.990773 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.990752    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lprjl"
Jul 31 01:37:23.991323 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.991297    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n6nsp\""
Jul 31 01:37:23.991599 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.991568    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 01:37:23.991683 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.991622    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 01:37:23.993174 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.993145    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:23.993272 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:23.993209    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:23.994380 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.994352    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 01:37:23.994473 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.994443    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fmdz4\""
Jul 31 01:37:23.994694 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.994674    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 01:37:23.994905 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.994888    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 01:37:23.995762 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.995740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-685q7"
Jul 31 01:37:23.998506 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.998479    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 01:37:23.998795 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.998774    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 01:37:23.999036 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.999020    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7dktv\""
Jul 31 01:37:23.999224 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:23.999209    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 01:37:24.000518 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.000480    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.003108 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.003089    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 01:37:24.003877 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.003829    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 01:37:24.004176 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.004156    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 01:37:24.004320 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.004301    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 01:37:24.004396 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.004377    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 01:37:24.004887 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.004863    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l7qbr\""
Jul 31 01:37:24.005099 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.005078    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 01:37:24.043529 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.043496    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c9fjv"
Jul 31 01:37:24.050425 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.050401    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c9fjv"
Jul 31 01:37:24.076170 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.076140    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 01:37:24.086519 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086492    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e26e8e31-84dc-4178-9f3d-172319dd06cb-konnectivity-ca\") pod \"konnectivity-agent-88pgt\" (UID: \"e26e8e31-84dc-4178-9f3d-172319dd06cb\") " pod="kube-system/konnectivity-agent-88pgt"
Jul 31 01:37:24.086682 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086534    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-sysctl-conf\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.086682 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086562    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b46eb842-87fe-4021-b184-0ff856e849ad-etc-tuned\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.086682 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086604    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-var-lib-cni-multus\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.086682 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086622    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:24.086682 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086647    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjwvr\" (UniqueName: \"kubernetes.io/projected/03ea1fd1-9975-468b-8297-41c0e36a9d00-kube-api-access-xjwvr\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.086682 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086672    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5c6c5a5-2a0c-48e2-b968-93837c492672-host\") pod \"node-ca-lprjl\" (UID: \"d5c6c5a5-2a0c-48e2-b968-93837c492672\") " pod="openshift-image-registry/node-ca-lprjl"
Jul 31 01:37:24.086954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086689    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ada9ca69-ceee-41dd-9594-41aaf46caf56-host-slash\") pod \"iptables-alerter-685q7\" (UID: \"ada9ca69-ceee-41dd-9594-41aaf46caf56\") " pod="openshift-network-operator/iptables-alerter-685q7"
Jul 31 01:37:24.086954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086706    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-kubelet\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.086954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086729    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-systemd-units\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.086954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086771    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-run-multus-certs\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.086954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086792    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.086954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086810    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-device-dir\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.086954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086827    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8525e366-5355-4406-8e20-88f336bf1e44-tmp-dir\") pod \"node-resolver-tqppw\" (UID: \"8525e366-5355-4406-8e20-88f336bf1e44\") " pod="openshift-dns/node-resolver-tqppw"
Jul 31 01:37:24.086954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086849    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45djd\" (UniqueName: \"kubernetes.io/projected/32bf4714-1aae-4a26-a0d3-cb99be970774-kube-api-access-45djd\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.086954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086871    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-sysconfig\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.086954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086888    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-kubernetes\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.086954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086911    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62fxz\" (UniqueName: \"kubernetes.io/projected/ada9ca69-ceee-41dd-9594-41aaf46caf56-kube-api-access-62fxz\") pod \"iptables-alerter-685q7\" (UID: \"ada9ca69-ceee-41dd-9594-41aaf46caf56\") " pod="openshift-network-operator/iptables-alerter-685q7"
Jul 31 01:37:24.086954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086931    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-run-netns\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086964    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-var-lib-openvswitch\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086979    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-run-ovn\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.086994    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-system-cni-dir\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087009    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-sysctl-d\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087037    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-multus-cni-dir\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087065    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-os-release\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087081    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-multus-socket-dir-parent\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087101    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-run-k8s-cni-cncf-io\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087117    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-etc-kubernetes\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087138    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwdpx\" (UniqueName: \"kubernetes.io/projected/8525e366-5355-4406-8e20-88f336bf1e44-kube-api-access-kwdpx\") pod \"node-resolver-tqppw\" (UID: \"8525e366-5355-4406-8e20-88f336bf1e44\") " pod="openshift-dns/node-resolver-tqppw"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087172    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrc8\" (UniqueName: \"kubernetes.io/projected/d5c6c5a5-2a0c-48e2-b968-93837c492672-kube-api-access-2hrc8\") pod \"node-ca-lprjl\" (UID: \"d5c6c5a5-2a0c-48e2-b968-93837c492672\") " pod="openshift-image-registry/node-ca-lprjl"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087198    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-cni-binary-copy\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087224    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-sys-fs\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087251    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-run\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087291    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32bf4714-1aae-4a26-a0d3-cb99be970774-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087328    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b46eb842-87fe-4021-b184-0ff856e849ad-tmp\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.087364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087348    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32e17105-e7bb-4ad2-8ef5-77660ff9a429-kubelet-config\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087364    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32e17105-e7bb-4ad2-8ef5-77660ff9a429-dbus\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087386    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-slash\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087411    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/efb771fd-993f-47ab-9e24-09976691283b-env-overrides\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087435    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32bf4714-1aae-4a26-a0d3-cb99be970774-cnibin\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087453    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087475    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-etc-openvswitch\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087496    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-cni-bin\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjkp5\" (UniqueName: \"kubernetes.io/projected/efb771fd-993f-47ab-9e24-09976691283b-kube-api-access-cjkp5\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087529    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-multus-conf-dir\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087544    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-lib-modules\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087560    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ada9ca69-ceee-41dd-9594-41aaf46caf56-iptables-alerter-script\") pod \"iptables-alerter-685q7\" (UID: \"ada9ca69-ceee-41dd-9594-41aaf46caf56\") " pod="openshift-network-operator/iptables-alerter-685q7"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087575    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-node-log\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087600    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-cnibin\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087615    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvxr\" (UniqueName: \"kubernetes.io/projected/064895ac-87e6-4098-9f8d-13b44c5c7a10-kube-api-access-gcvxr\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087634    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-run-systemd\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087675    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32bf4714-1aae-4a26-a0d3-cb99be970774-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.088056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087695    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e26e8e31-84dc-4178-9f3d-172319dd06cb-agent-certs\") pod \"konnectivity-agent-88pgt\" (UID: \"e26e8e31-84dc-4178-9f3d-172319dd06cb\") " pod="kube-system/konnectivity-agent-88pgt"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087730    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-var-lib-kubelet\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087754    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-cni-netd\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087773    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-run-netns\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087792    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-modprobe-d\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087816    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32bf4714-1aae-4a26-a0d3-cb99be970774-os-release\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087855    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32bf4714-1aae-4a26-a0d3-cb99be970774-cni-binary-copy\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-registration-dir\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087904    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzwgn\" (UniqueName: \"kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn\") pod \"network-check-target-vv7wq\" (UID: \"346810b7-b42a-44e7-8e2f-d1e7942fbe20\") " pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087922    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/efb771fd-993f-47ab-9e24-09976691283b-ovnkube-config\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087940    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087956    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-systemd\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087978    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-sys\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.087999    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-host\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088017    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/efb771fd-993f-47ab-9e24-09976691283b-ovnkube-script-lib\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088036    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-multus-daemon-config\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088071    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/efb771fd-993f-47ab-9e24-09976691283b-ovn-node-metrics-cert\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.088708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088103    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-var-lib-cni-bin\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088120    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-hostroot\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088135    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8525e366-5355-4406-8e20-88f336bf1e44-hosts-file\") pod \"node-resolver-tqppw\" (UID: \"8525e366-5355-4406-8e20-88f336bf1e44\") " pod="openshift-dns/node-resolver-tqppw"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088157    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32bf4714-1aae-4a26-a0d3-cb99be970774-system-cni-dir\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088192    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-socket-dir\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088212    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088231    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbsq\" (UniqueName: \"kubernetes.io/projected/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-kube-api-access-lmbsq\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088253    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32bf4714-1aae-4a26-a0d3-cb99be970774-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088273    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjdg\" (UniqueName: \"kubernetes.io/projected/b46eb842-87fe-4021-b184-0ff856e849ad-kube-api-access-nvjdg\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088290    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-run-openvswitch\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088318    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-log-socket\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088343    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088368    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5c6c5a5-2a0c-48e2-b968-93837c492672-serviceca\") pod \"node-ca-lprjl\" (UID: \"d5c6c5a5-2a0c-48e2-b968-93837c492672\") " pod="openshift-image-registry/node-ca-lprjl"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088385    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-var-lib-kubelet\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.088951    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e26e8e31-84dc-4178-9f3d-172319dd06cb-konnectivity-ca\") pod \"konnectivity-agent-88pgt\" (UID: \"e26e8e31-84dc-4178-9f3d-172319dd06cb\") " pod="kube-system/konnectivity-agent-88pgt"
Jul 31 01:37:24.089309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.089121    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-sysctl-conf\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.089934 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.089413    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 01:37:24.089934 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.089765    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-device-dir\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.090668 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.090005    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-sysconfig\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.090668 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.090061    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-kubernetes\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.090668 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.090139    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.090668 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.090231    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-sysctl-d\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.090668 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.090257    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-lib-modules\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.090668 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.090354    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-sys-fs\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.090668 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.090405    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-run\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.090668 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.090442    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32bf4714-1aae-4a26-a0d3-cb99be970774-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.091207 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.090853    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32bf4714-1aae-4a26-a0d3-cb99be970774-system-cni-dir\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.091207 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.090922    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-systemd\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.091207 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.090975    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-sys\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.091207 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.091025    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-host\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.091207 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.091158    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-var-lib-kubelet\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.091456 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.091439    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32bf4714-1aae-4a26-a0d3-cb99be970774-cnibin\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.091513 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.091493    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b46eb842-87fe-4021-b184-0ff856e849ad-etc-modprobe-d\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.091568 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.091522    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32bf4714-1aae-4a26-a0d3-cb99be970774-os-release\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.091645 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.091565    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32bf4714-1aae-4a26-a0d3-cb99be970774-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.091708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.091692    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-socket-dir\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.091768 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.091751    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-registration-dir\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.091911 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.091828    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03ea1fd1-9975-468b-8297-41c0e36a9d00-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.092059 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.092033    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32bf4714-1aae-4a26-a0d3-cb99be970774-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.092134 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.092079    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32bf4714-1aae-4a26-a0d3-cb99be970774-cni-binary-copy\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.094160 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.094133    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b46eb842-87fe-4021-b184-0ff856e849ad-tmp\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.094364 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.094345    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e26e8e31-84dc-4178-9f3d-172319dd06cb-agent-certs\") pod \"konnectivity-agent-88pgt\" (UID: \"e26e8e31-84dc-4178-9f3d-172319dd06cb\") " pod="kube-system/konnectivity-agent-88pgt"
Jul 31 01:37:24.094863 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.094847    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b46eb842-87fe-4021-b184-0ff856e849ad-etc-tuned\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.099939 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.099871    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjdg\" (UniqueName: \"kubernetes.io/projected/b46eb842-87fe-4021-b184-0ff856e849ad-kube-api-access-nvjdg\") pod \"tuned-mvmpt\" (UID: \"b46eb842-87fe-4021-b184-0ff856e849ad\") " pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.100675 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.100628    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45djd\" (UniqueName: \"kubernetes.io/projected/32bf4714-1aae-4a26-a0d3-cb99be970774-kube-api-access-45djd\") pod \"multus-additional-cni-plugins-tq27z\" (UID: \"32bf4714-1aae-4a26-a0d3-cb99be970774\") " pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.100912 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.100880    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjwvr\" (UniqueName: \"kubernetes.io/projected/03ea1fd1-9975-468b-8297-41c0e36a9d00-kube-api-access-xjwvr\") pod \"aws-ebs-csi-driver-node-xbhpj\" (UID: \"03ea1fd1-9975-468b-8297-41c0e36a9d00\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189344    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwdpx\" (UniqueName: \"kubernetes.io/projected/8525e366-5355-4406-8e20-88f336bf1e44-kube-api-access-kwdpx\") pod \"node-resolver-tqppw\" (UID: \"8525e366-5355-4406-8e20-88f336bf1e44\") " pod="openshift-dns/node-resolver-tqppw"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189391    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrc8\" (UniqueName: \"kubernetes.io/projected/d5c6c5a5-2a0c-48e2-b968-93837c492672-kube-api-access-2hrc8\") pod \"node-ca-lprjl\" (UID: \"d5c6c5a5-2a0c-48e2-b968-93837c492672\") " pod="openshift-image-registry/node-ca-lprjl"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189408    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-cni-binary-copy\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189428    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32e17105-e7bb-4ad2-8ef5-77660ff9a429-kubelet-config\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189635    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32e17105-e7bb-4ad2-8ef5-77660ff9a429-dbus\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189676    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-slash\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189705    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/efb771fd-993f-47ab-9e24-09976691283b-env-overrides\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189734    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-etc-openvswitch\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189759    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-cni-bin\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189785    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjkp5\" (UniqueName: \"kubernetes.io/projected/efb771fd-993f-47ab-9e24-09976691283b-kube-api-access-cjkp5\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189810    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-multus-conf-dir\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189839    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ada9ca69-ceee-41dd-9594-41aaf46caf56-iptables-alerter-script\") pod \"iptables-alerter-685q7\" (UID: \"ada9ca69-ceee-41dd-9594-41aaf46caf56\") " pod="openshift-network-operator/iptables-alerter-685q7"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189865    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-node-log\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189889    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-cnibin\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189915    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-etc-openvswitch\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189926    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-cni-binary-copy\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.190486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189934    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvxr\" (UniqueName: \"kubernetes.io/projected/064895ac-87e6-4098-9f8d-13b44c5c7a10-kube-api-access-gcvxr\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189960    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-run-systemd\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189977    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-multus-conf-dir\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.189990    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-cni-netd\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190005    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-cni-bin\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190016    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-run-netns\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190049    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwgn\" (UniqueName: \"kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn\") pod \"network-check-target-vv7wq\" (UID: \"346810b7-b42a-44e7-8e2f-d1e7942fbe20\") " pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190052    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32e17105-e7bb-4ad2-8ef5-77660ff9a429-dbus\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190075    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/efb771fd-993f-47ab-9e24-09976691283b-ovnkube-config\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190108    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/efb771fd-993f-47ab-9e24-09976691283b-ovnkube-script-lib\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190113    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-slash\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190132    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-multus-daemon-config\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190167    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/efb771fd-993f-47ab-9e24-09976691283b-ovn-node-metrics-cert\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190185    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-var-lib-cni-bin\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190199    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32e17105-e7bb-4ad2-8ef5-77660ff9a429-kubelet-config\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190201    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-hostroot\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190222    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-hostroot\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190226    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8525e366-5355-4406-8e20-88f336bf1e44-hosts-file\") pod \"node-resolver-tqppw\" (UID: \"8525e366-5355-4406-8e20-88f336bf1e44\") " pod="openshift-dns/node-resolver-tqppw"
Jul 31 01:37:24.191465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190250    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190258    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8525e366-5355-4406-8e20-88f336bf1e44-hosts-file\") pod \"node-resolver-tqppw\" (UID: \"8525e366-5355-4406-8e20-88f336bf1e44\") " pod="openshift-dns/node-resolver-tqppw"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190273    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbsq\" (UniqueName: \"kubernetes.io/projected/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-kube-api-access-lmbsq\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190292    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-run-openvswitch\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190310    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-log-socket\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190337    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190365    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5c6c5a5-2a0c-48e2-b968-93837c492672-serviceca\") pod \"node-ca-lprjl\" (UID: \"d5c6c5a5-2a0c-48e2-b968-93837c492672\") " pod="openshift-image-registry/node-ca-lprjl"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190389    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-var-lib-kubelet\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.191511    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-var-lib-cni-multus\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.190419    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.191595    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190705    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-run-openvswitch\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190690    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.191667    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret podName:32e17105-e7bb-4ad2-8ef5-77660ff9a429 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:24.691620892 +0000 UTC m=+3.057366524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret") pod "global-pull-secret-syncer-dnk4v" (UID: "32e17105-e7bb-4ad2-8ef5-77660ff9a429") : object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190634    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-var-lib-kubelet\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190795    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-var-lib-cni-bin\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190480    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-log-socket\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.192355 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190830    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-cni-netd\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190836    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-run-netns\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190872    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-cnibin\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190949    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/efb771fd-993f-47ab-9e24-09976691283b-env-overrides\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190784    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ada9ca69-ceee-41dd-9594-41aaf46caf56-iptables-alerter-script\") pod \"iptables-alerter-685q7\" (UID: \"ada9ca69-ceee-41dd-9594-41aaf46caf56\") " pod="openshift-network-operator/iptables-alerter-685q7"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.191016    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-run-systemd\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.191380    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5c6c5a5-2a0c-48e2-b968-93837c492672-serviceca\") pod \"node-ca-lprjl\" (UID: \"d5c6c5a5-2a0c-48e2-b968-93837c492672\") " pod="openshift-image-registry/node-ca-lprjl"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.191770    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-var-lib-cni-multus\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190814    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-node-log\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192050    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/efb771fd-993f-47ab-9e24-09976691283b-ovnkube-config\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.190753    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-multus-daemon-config\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.191703    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5c6c5a5-2a0c-48e2-b968-93837c492672-host\") pod \"node-ca-lprjl\" (UID: \"d5c6c5a5-2a0c-48e2-b968-93837c492672\") " pod="openshift-image-registry/node-ca-lprjl"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.192113    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192146    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ada9ca69-ceee-41dd-9594-41aaf46caf56-host-slash\") pod \"iptables-alerter-685q7\" (UID: \"ada9ca69-ceee-41dd-9594-41aaf46caf56\") " pod="openshift-network-operator/iptables-alerter-685q7"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192191    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-kubelet\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192221    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-systemd-units\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192268    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-run-multus-certs\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.192300    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs podName:064895ac-87e6-4098-9f8d-13b44c5c7a10 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:24.6922654 +0000 UTC m=+3.058011011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs") pod "network-metrics-daemon-x94zh" (UID: "064895ac-87e6-4098-9f8d-13b44c5c7a10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:24.193168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192333    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-run-multus-certs\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192348    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192384    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8525e366-5355-4406-8e20-88f336bf1e44-tmp-dir\") pod \"node-resolver-tqppw\" (UID: \"8525e366-5355-4406-8e20-88f336bf1e44\") " pod="openshift-dns/node-resolver-tqppw"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192392    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ada9ca69-ceee-41dd-9594-41aaf46caf56-host-slash\") pod \"iptables-alerter-685q7\" (UID: \"ada9ca69-ceee-41dd-9594-41aaf46caf56\") " pod="openshift-network-operator/iptables-alerter-685q7"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192429    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62fxz\" (UniqueName: \"kubernetes.io/projected/ada9ca69-ceee-41dd-9594-41aaf46caf56-kube-api-access-62fxz\") pod \"iptables-alerter-685q7\" (UID: \"ada9ca69-ceee-41dd-9594-41aaf46caf56\") " pod="openshift-network-operator/iptables-alerter-685q7"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192448    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-kubelet\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192461    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-run-netns\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192494    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-var-lib-openvswitch\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192523    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-run-ovn\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192528    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/efb771fd-993f-47ab-9e24-09976691283b-ovnkube-script-lib\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192557    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-system-cni-dir\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192588    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-run-netns\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192600    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-run-ovn\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192590    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-multus-cni-dir\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192673    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-os-release\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192746    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-multus-socket-dir-parent\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192775    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-var-lib-openvswitch\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-run-k8s-cni-cncf-io\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.194063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192523    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.194914 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.192953    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8525e366-5355-4406-8e20-88f336bf1e44-tmp-dir\") pod \"node-resolver-tqppw\" (UID: \"8525e366-5355-4406-8e20-88f336bf1e44\") " pod="openshift-dns/node-resolver-tqppw"
Jul 31 01:37:24.194914 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.193238    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-host-run-k8s-cni-cncf-io\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.194914 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.193246    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/efb771fd-993f-47ab-9e24-09976691283b-systemd-units\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.194914 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.193311    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-os-release\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.194914 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.193372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-etc-kubernetes\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.194914 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.193424    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-system-cni-dir\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.194914 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.193375    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5c6c5a5-2a0c-48e2-b968-93837c492672-host\") pod \"node-ca-lprjl\" (UID: \"d5c6c5a5-2a0c-48e2-b968-93837c492672\") " pod="openshift-image-registry/node-ca-lprjl"
Jul 31 01:37:24.194914 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.193482    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-etc-kubernetes\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.194914 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.193490    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-multus-socket-dir-parent\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.194914 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.193552    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-multus-cni-dir\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.196428 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.196388    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/efb771fd-993f-47ab-9e24-09976691283b-ovn-node-metrics-cert\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.200452 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.200178    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 01:37:24.200452 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.200204    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 01:37:24.200452 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.200220    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gzwgn for pod openshift-network-diagnostics/network-check-target-vv7wq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:24.200452 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.200282    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn podName:346810b7-b42a-44e7-8e2f-d1e7942fbe20 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:24.700263057 +0000 UTC m=+3.066008665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwgn" (UniqueName: "kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn") pod "network-check-target-vv7wq" (UID: "346810b7-b42a-44e7-8e2f-d1e7942fbe20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:24.202161 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.202140    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjkp5\" (UniqueName: \"kubernetes.io/projected/efb771fd-993f-47ab-9e24-09976691283b-kube-api-access-cjkp5\") pod \"ovnkube-node-fsprs\" (UID: \"efb771fd-993f-47ab-9e24-09976691283b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.202468 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.202426    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62fxz\" (UniqueName: \"kubernetes.io/projected/ada9ca69-ceee-41dd-9594-41aaf46caf56-kube-api-access-62fxz\") pod \"iptables-alerter-685q7\" (UID: \"ada9ca69-ceee-41dd-9594-41aaf46caf56\") " pod="openshift-network-operator/iptables-alerter-685q7"
Jul 31 01:37:24.202695 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.202672    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwdpx\" (UniqueName: \"kubernetes.io/projected/8525e366-5355-4406-8e20-88f336bf1e44-kube-api-access-kwdpx\") pod \"node-resolver-tqppw\" (UID: \"8525e366-5355-4406-8e20-88f336bf1e44\") " pod="openshift-dns/node-resolver-tqppw"
Jul 31 01:37:24.203158 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.203133    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrc8\" (UniqueName: \"kubernetes.io/projected/d5c6c5a5-2a0c-48e2-b968-93837c492672-kube-api-access-2hrc8\") pod \"node-ca-lprjl\" (UID: \"d5c6c5a5-2a0c-48e2-b968-93837c492672\") " pod="openshift-image-registry/node-ca-lprjl"
Jul 31 01:37:24.203435 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.203415    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvxr\" (UniqueName: \"kubernetes.io/projected/064895ac-87e6-4098-9f8d-13b44c5c7a10-kube-api-access-gcvxr\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:24.204406 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.204387    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbsq\" (UniqueName: \"kubernetes.io/projected/1c03aedf-0207-4282-8b2f-8ab2a724e9f1-kube-api-access-lmbsq\") pod \"multus-5l5wj\" (UID: \"1c03aedf-0207-4282-8b2f-8ab2a724e9f1\") " pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.282025 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.281930    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj"
Jul 31 01:37:24.291591 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.291550    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-88pgt"
Jul 31 01:37:24.303745 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.303710    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mvmpt"
Jul 31 01:37:24.311227 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.311203    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tq27z"
Jul 31 01:37:24.322457 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.322432    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5l5wj"
Jul 31 01:37:24.330519 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.330498    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tqppw"
Jul 31 01:37:24.339501 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.339481    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lprjl"
Jul 31 01:37:24.351423 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.351387    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-685q7"
Jul 31 01:37:24.358552 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.358534    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:24.450701 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.450662    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 01:37:24.697752 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.697710    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:24.697964 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.697761    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:24.697964 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.697885    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:24.697964 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.697890    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:24.697964 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.697955    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs podName:064895ac-87e6-4098-9f8d-13b44c5c7a10 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:25.697936118 +0000 UTC m=+4.063681750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs") pod "network-metrics-daemon-x94zh" (UID: "064895ac-87e6-4098-9f8d-13b44c5c7a10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:24.698161 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.697972    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret podName:32e17105-e7bb-4ad2-8ef5-77660ff9a429 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:25.697963713 +0000 UTC m=+4.063709321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret") pod "global-pull-secret-syncer-dnk4v" (UID: "32e17105-e7bb-4ad2-8ef5-77660ff9a429") : object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:24.798973 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:24.798919    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwgn\" (UniqueName: \"kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn\") pod \"network-check-target-vv7wq\" (UID: \"346810b7-b42a-44e7-8e2f-d1e7942fbe20\") " pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:24.799160 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.799122    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 01:37:24.799160 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.799154    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 01:37:24.799269 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.799167    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gzwgn for pod openshift-network-diagnostics/network-check-target-vv7wq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:24.799269 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:24.799231    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn podName:346810b7-b42a-44e7-8e2f-d1e7942fbe20 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:25.7992106 +0000 UTC m=+4.164956228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwgn" (UniqueName: "kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn") pod "network-check-target-vv7wq" (UID: "346810b7-b42a-44e7-8e2f-d1e7942fbe20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:25.054626 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:25.054530    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 01:32:24 +0000 UTC" deadline="2028-01-15 20:57:14.520762282 +0000 UTC"
Jul 31 01:37:25.054626 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:25.054580    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12811h19m49.466188444s"
Jul 31 01:37:25.145633 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:25.145599    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:25.145826 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:25.145730    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:25.212231 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:25.212197    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26e8e31_84dc_4178_9f3d_172319dd06cb.slice/crio-9db2b31df9926fb291e72d56410f31ef424528d376140f025a41099bd31ff778 WatchSource:0}: Error finding container 9db2b31df9926fb291e72d56410f31ef424528d376140f025a41099bd31ff778: Status 404 returned error can't find the container with id 9db2b31df9926fb291e72d56410f31ef424528d376140f025a41099bd31ff778
Jul 31 01:37:25.217765 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:25.217737    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb771fd_993f_47ab_9e24_09976691283b.slice/crio-85e4ba5e8056a532ab57e543f5e2918bee440321b72a7c495edb030ba4ed9cf4 WatchSource:0}: Error finding container 85e4ba5e8056a532ab57e543f5e2918bee440321b72a7c495edb030ba4ed9cf4: Status 404 returned error can't find the container with id 85e4ba5e8056a532ab57e543f5e2918bee440321b72a7c495edb030ba4ed9cf4
Jul 31 01:37:25.219193 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:25.219101    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c6c5a5_2a0c_48e2_b968_93837c492672.slice/crio-a594e9f9cd0c65e16053d4f5149d880922c8635fb44130fcfcbeb3ab9798a8a4 WatchSource:0}: Error finding container a594e9f9cd0c65e16053d4f5149d880922c8635fb44130fcfcbeb3ab9798a8a4: Status 404 returned error can't find the container with id a594e9f9cd0c65e16053d4f5149d880922c8635fb44130fcfcbeb3ab9798a8a4
Jul 31 01:37:25.220769 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:25.220379    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8525e366_5355_4406_8e20_88f336bf1e44.slice/crio-44b7e1c9cb6a45479f88cad4ffe31ecb2d2f35e51121978b4dce15059ff50ac9 WatchSource:0}: Error finding container 44b7e1c9cb6a45479f88cad4ffe31ecb2d2f35e51121978b4dce15059ff50ac9: Status 404 returned error can't find the container with id 44b7e1c9cb6a45479f88cad4ffe31ecb2d2f35e51121978b4dce15059ff50ac9
Jul 31 01:37:25.708808 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:25.708434    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:25.709025 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:25.708863    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:25.709099 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:25.709055    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:25.709154 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:25.709122    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs podName:064895ac-87e6-4098-9f8d-13b44c5c7a10 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:27.709100738 +0000 UTC m=+6.074846351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs") pod "network-metrics-daemon-x94zh" (UID: "064895ac-87e6-4098-9f8d-13b44c5c7a10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:25.709602 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:25.709580    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:25.709791 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:25.709662    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret podName:32e17105-e7bb-4ad2-8ef5-77660ff9a429 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:27.709628611 +0000 UTC m=+6.075374226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret") pod "global-pull-secret-syncer-dnk4v" (UID: "32e17105-e7bb-4ad2-8ef5-77660ff9a429") : object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:25.811463 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:25.811419    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwgn\" (UniqueName: \"kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn\") pod \"network-check-target-vv7wq\" (UID: \"346810b7-b42a-44e7-8e2f-d1e7942fbe20\") " pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:25.811658 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:25.811615    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 01:37:25.811742 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:25.811677    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 01:37:25.811742 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:25.811694    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gzwgn for pod openshift-network-diagnostics/network-check-target-vv7wq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:25.811851 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:25.811760    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn podName:346810b7-b42a-44e7-8e2f-d1e7942fbe20 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:27.811739156 +0000 UTC m=+6.177484782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwgn" (UniqueName: "kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn") pod "network-check-target-vv7wq" (UID: "346810b7-b42a-44e7-8e2f-d1e7942fbe20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:26.056280 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.055504    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 01:32:24 +0000 UTC" deadline="2028-03-22 17:03:18.679438326 +0000 UTC"
Jul 31 01:37:26.056280 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.055545    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14415h25m52.623897479s"
Jul 31 01:37:26.148503 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.148468    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:26.148721 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:26.148615    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:26.149214 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.149076    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:26.149214 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:26.149172    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:26.163843 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.163073    2556 generic.go:356] "Generic (PLEG): container finished" podID="6a631120ad11f3f853e9f18ea275f194" containerID="4a6cc4bfc3f67d1ae53fbb40aaef6829a7ed44907400c920bb2a615f15ae4a2d" exitCode=0
Jul 31 01:37:26.163843 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.163168    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal" event={"ID":"6a631120ad11f3f853e9f18ea275f194","Type":"ContainerDied","Data":"4a6cc4bfc3f67d1ae53fbb40aaef6829a7ed44907400c920bb2a615f15ae4a2d"}
Jul 31 01:37:26.175289 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.174542    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal" event={"ID":"f923c99785cedeeb9d5a1e0081b46a58","Type":"ContainerStarted","Data":"eb8885a4ffa90b8d88393711ffef99375317cdc1ed0a4eeb2ac7ac66d84ddb09"}
Jul 31 01:37:26.186805 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.186729    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mvmpt" event={"ID":"b46eb842-87fe-4021-b184-0ff856e849ad","Type":"ContainerStarted","Data":"af4bc40fb9676eab8573d2377dd773fd7fa2a73e25bc37bedcbf5dc3e719000a"}
Jul 31 01:37:26.196306 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.196247    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lprjl" event={"ID":"d5c6c5a5-2a0c-48e2-b968-93837c492672","Type":"ContainerStarted","Data":"a594e9f9cd0c65e16053d4f5149d880922c8635fb44130fcfcbeb3ab9798a8a4"}
Jul 31 01:37:26.199022 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.198914    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs" event={"ID":"efb771fd-993f-47ab-9e24-09976691283b","Type":"ContainerStarted","Data":"85e4ba5e8056a532ab57e543f5e2918bee440321b72a7c495edb030ba4ed9cf4"}
Jul 31 01:37:26.203870 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.203802    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-88pgt" event={"ID":"e26e8e31-84dc-4178-9f3d-172319dd06cb","Type":"ContainerStarted","Data":"9db2b31df9926fb291e72d56410f31ef424528d376140f025a41099bd31ff778"}
Jul 31 01:37:26.208039 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.207919    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-685q7" event={"ID":"ada9ca69-ceee-41dd-9594-41aaf46caf56","Type":"ContainerStarted","Data":"5740b411a43e82e952bcf224edb86f7a7f3aa9f0a1463481daf84afa18875843"}
Jul 31 01:37:26.233567 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.233459    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5l5wj" event={"ID":"1c03aedf-0207-4282-8b2f-8ab2a724e9f1","Type":"ContainerStarted","Data":"d2eeacc23d6d181b52b560698886e48e146e70a256e13ce88458601f222ae7a2"}
Jul 31 01:37:26.237974 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.237943    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq27z" event={"ID":"32bf4714-1aae-4a26-a0d3-cb99be970774","Type":"ContainerStarted","Data":"9f78d2866b9d6a28e824eace8127adee4f08f8fb8638186b02d6e14802d96917"}
Jul 31 01:37:26.240736 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.240708    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj" event={"ID":"03ea1fd1-9975-468b-8297-41c0e36a9d00","Type":"ContainerStarted","Data":"7c8c6851ba8edc85ceb82edbd262c8f5a3bae1cb70e7ad8394374ee6d44113d6"}
Jul 31 01:37:26.254065 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:26.254038    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tqppw" event={"ID":"8525e366-5355-4406-8e20-88f336bf1e44","Type":"ContainerStarted","Data":"44b7e1c9cb6a45479f88cad4ffe31ecb2d2f35e51121978b4dce15059ff50ac9"}
Jul 31 01:37:27.146158 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:27.146125    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:27.146695 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:27.146261    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:27.268714 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:27.268675    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal" event={"ID":"6a631120ad11f3f853e9f18ea275f194","Type":"ContainerStarted","Data":"dbe042038f694d2c27c2be16bd02d6ea9e7d865d64159d5b78c674fac7568008"}
Jul 31 01:37:27.283408 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:27.283342    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal" podStartSLOduration=4.283316508 podStartE2EDuration="4.283316508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:37:26.196130095 +0000 UTC m=+4.561875727" watchObservedRunningTime="2026-07-31 01:37:27.283316508 +0000 UTC m=+5.649062142"
Jul 31 01:37:27.727851 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:27.727806    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:27.728061 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:27.727869    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:27.728061 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:27.728014    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:27.728174 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:27.728081    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs podName:064895ac-87e6-4098-9f8d-13b44c5c7a10 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:31.728059294 +0000 UTC m=+10.093804916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs") pod "network-metrics-daemon-x94zh" (UID: "064895ac-87e6-4098-9f8d-13b44c5c7a10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:27.728607 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:27.728550    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:27.728607 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:27.728610    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret podName:32e17105-e7bb-4ad2-8ef5-77660ff9a429 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:31.728590912 +0000 UTC m=+10.094336607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret") pod "global-pull-secret-syncer-dnk4v" (UID: "32e17105-e7bb-4ad2-8ef5-77660ff9a429") : object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:27.829530 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:27.828754    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwgn\" (UniqueName: \"kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn\") pod \"network-check-target-vv7wq\" (UID: \"346810b7-b42a-44e7-8e2f-d1e7942fbe20\") " pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:27.829530 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:27.829010    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 01:37:27.829530 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:27.829037    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 01:37:27.829530 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:27.829050    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gzwgn for pod openshift-network-diagnostics/network-check-target-vv7wq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:27.829530 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:27.829114    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn podName:346810b7-b42a-44e7-8e2f-d1e7942fbe20 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:31.829094651 +0000 UTC m=+10.194840263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwgn" (UniqueName: "kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn") pod "network-check-target-vv7wq" (UID: "346810b7-b42a-44e7-8e2f-d1e7942fbe20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:28.147773 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:28.147738    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:28.148265 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:28.147899    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:28.148265 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:28.147972    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:28.148265 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:28.148045    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:29.145230 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:29.145200    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:29.145443 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:29.145337    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:30.148119 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:30.148085    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:30.148546 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:30.148246    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:30.148546 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:30.148528    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:30.148660 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:30.148621    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:31.146058 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:31.146026    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:31.146253 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:31.146212    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:31.763354 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:31.763318    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:31.763858 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:31.763404    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:31.763858 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:31.763526    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:31.763858 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:31.763605    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret podName:32e17105-e7bb-4ad2-8ef5-77660ff9a429 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:39.763585329 +0000 UTC m=+18.129330935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret") pod "global-pull-secret-syncer-dnk4v" (UID: "32e17105-e7bb-4ad2-8ef5-77660ff9a429") : object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:31.763858 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:31.763680    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:31.763858 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:31.763741    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs podName:064895ac-87e6-4098-9f8d-13b44c5c7a10 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:39.763726794 +0000 UTC m=+18.129472402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs") pod "network-metrics-daemon-x94zh" (UID: "064895ac-87e6-4098-9f8d-13b44c5c7a10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:31.864245 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:31.864205    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwgn\" (UniqueName: \"kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn\") pod \"network-check-target-vv7wq\" (UID: \"346810b7-b42a-44e7-8e2f-d1e7942fbe20\") " pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:31.864458 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:31.864437    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 01:37:31.864458 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:31.864461    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 01:37:31.864661 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:31.864473    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gzwgn for pod openshift-network-diagnostics/network-check-target-vv7wq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:31.864661 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:31.864537    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn podName:346810b7-b42a-44e7-8e2f-d1e7942fbe20 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:39.864518048 +0000 UTC m=+18.230263653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwgn" (UniqueName: "kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn") pod "network-check-target-vv7wq" (UID: "346810b7-b42a-44e7-8e2f-d1e7942fbe20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:32.147544 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:32.147517    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:32.147745 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:32.147635    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:32.147964 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:32.147947    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:32.148054 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:32.148028    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:33.146121 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:33.146091    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:33.146586 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:33.146256    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:34.145482 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:34.145446    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:34.145698 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:34.145493    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:34.145698 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:34.145623    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:34.145827 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:34.145741    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:35.145852 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:35.145823    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:35.146301 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:35.145953    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:36.145273 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:36.145236    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:36.145448 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:36.145243    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:36.145448 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:36.145376    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:36.145588 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:36.145440    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:37.146070 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:37.146038    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:37.146495 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:37.146144    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:38.145998 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:38.145961    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:38.146191 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:38.146156    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:38.146544 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:38.146223    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:38.146544 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:38.146331    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:39.145429 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:39.145397    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:39.145683 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:39.145512    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:39.824990 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:39.824956    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:39.825459 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:39.825042    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:39.825459 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:39.825123    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:39.825459 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:39.825129    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:39.825459 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:39.825180    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret podName:32e17105-e7bb-4ad2-8ef5-77660ff9a429 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:55.825162414 +0000 UTC m=+34.190908020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret") pod "global-pull-secret-syncer-dnk4v" (UID: "32e17105-e7bb-4ad2-8ef5-77660ff9a429") : object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:39.825459 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:39.825200    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs podName:064895ac-87e6-4098-9f8d-13b44c5c7a10 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:55.825190432 +0000 UTC m=+34.190936043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs") pod "network-metrics-daemon-x94zh" (UID: "064895ac-87e6-4098-9f8d-13b44c5c7a10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:39.925965 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:39.925926    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwgn\" (UniqueName: \"kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn\") pod \"network-check-target-vv7wq\" (UID: \"346810b7-b42a-44e7-8e2f-d1e7942fbe20\") " pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:39.926132 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:39.926109    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 01:37:39.926206 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:39.926134    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 01:37:39.926206 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:39.926148    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gzwgn for pod openshift-network-diagnostics/network-check-target-vv7wq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:39.926305 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:39.926220    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn podName:346810b7-b42a-44e7-8e2f-d1e7942fbe20 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:55.926198671 +0000 UTC m=+34.291944292 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwgn" (UniqueName: "kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn") pod "network-check-target-vv7wq" (UID: "346810b7-b42a-44e7-8e2f-d1e7942fbe20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:40.147320 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:40.147236    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:40.147479 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:40.147356    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:40.147479 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:40.147463    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:40.147613 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:40.147577    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:41.145435 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:41.145404    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:41.145837 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:41.145514    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:42.146846 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:42.146810    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:42.147309 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:42.146943    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:42.147309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:42.147006    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:42.147309 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:42.147128    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:43.145244 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:43.145206    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:43.145417 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:43.145345    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:44.145642 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:44.145612    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:44.146225 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:44.145652    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:44.146225 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:44.145725    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:44.146225 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:44.145840    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:45.146400 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.146089    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:45.147159 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:45.146501    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:45.308552 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.308520    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lprjl" event={"ID":"d5c6c5a5-2a0c-48e2-b968-93837c492672","Type":"ContainerStarted","Data":"91fd4992cecade5e7f67c1869d6d2738c41795d407f0c52229e51b3b30581154"}
Jul 31 01:37:45.312137 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.312111    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs" event={"ID":"efb771fd-993f-47ab-9e24-09976691283b","Type":"ContainerStarted","Data":"adcb5f02f1672772cbad04e3df4fd3678287424af79f3e1b8752618a2e50bfa6"}
Jul 31 01:37:45.312274 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.312191    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs" event={"ID":"efb771fd-993f-47ab-9e24-09976691283b","Type":"ContainerStarted","Data":"aa2a2452b75c9fe589c8db820f9111e8d9f89ccb813e59897ac59d899dcd0775"}
Jul 31 01:37:45.312274 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.312239    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs" event={"ID":"efb771fd-993f-47ab-9e24-09976691283b","Type":"ContainerStarted","Data":"946fe8224cc2e7501cb7f13c72d7b4fe9e3ef76f30657565319ca67c210f505d"}
Jul 31 01:37:45.312386 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.312306    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs" event={"ID":"efb771fd-993f-47ab-9e24-09976691283b","Type":"ContainerStarted","Data":"67b001e600c8755068a075e79c5903f52b0a958b910cfb03ddc62a202c13e353"}
Jul 31 01:37:45.312443 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.312383    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs" event={"ID":"efb771fd-993f-47ab-9e24-09976691283b","Type":"ContainerStarted","Data":"f5b0b623fbba887f6423c6751342fdc1ac63c90ba185c505f0cb302fd4efcd4b"}
Jul 31 01:37:45.312443 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.312433    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs" event={"ID":"efb771fd-993f-47ab-9e24-09976691283b","Type":"ContainerStarted","Data":"b99a4af0901325bb862122e8c8a152777f77d9d7e179efa721d12c2095d369fd"}
Jul 31 01:37:45.313807 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.313781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-88pgt" event={"ID":"e26e8e31-84dc-4178-9f3d-172319dd06cb","Type":"ContainerStarted","Data":"da6bfaecd1158a735e3e8d8089b16248c157a4949f87cfe3c6b99b677bd7461c"}
Jul 31 01:37:45.315813 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.315789    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5l5wj" event={"ID":"1c03aedf-0207-4282-8b2f-8ab2a724e9f1","Type":"ContainerStarted","Data":"85a3178e0e758451f6fd3a8a6b695897b020a064cea0af8dea88c1c9aabff253"}
Jul 31 01:37:45.317860 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.317812    2556 generic.go:356] "Generic (PLEG): container finished" podID="32bf4714-1aae-4a26-a0d3-cb99be970774" containerID="f1eae9837291cc63f414f9f1b0f1461ceea473d22d51b1ee8154eba9acdb0947" exitCode=0
Jul 31 01:37:45.317976 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.317902    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq27z" event={"ID":"32bf4714-1aae-4a26-a0d3-cb99be970774","Type":"ContainerDied","Data":"f1eae9837291cc63f414f9f1b0f1461ceea473d22d51b1ee8154eba9acdb0947"}
Jul 31 01:37:45.319620 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.319502    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj" event={"ID":"03ea1fd1-9975-468b-8297-41c0e36a9d00","Type":"ContainerStarted","Data":"4a267122367217cec4c72fda085d89282413856a87d5a34cc33dd45b7f66b41f"}
Jul 31 01:37:45.321634 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.321612    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tqppw" event={"ID":"8525e366-5355-4406-8e20-88f336bf1e44","Type":"ContainerStarted","Data":"6a36c5684644e1ccf6b2dffc6f30ccb58ef44adcc7812fad127a0fb18dbb3de2"}
Jul 31 01:37:45.322378 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.322331    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lprjl" podStartSLOduration=4.024140033 podStartE2EDuration="23.322314167s" totalImagesPullingTime="19.298174134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:37:45.321698664 +0000 UTC m=+23.687444293" watchObservedRunningTime="2026-07-31 01:37:45.322314167 +0000 UTC m=+23.688059796"
Jul 31 01:37:45.322468 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.322420    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal" podStartSLOduration=22.322412371 podStartE2EDuration="22.322412371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:37:27.284668463 +0000 UTC m=+5.650414099" watchObservedRunningTime="2026-07-31 01:37:45.322412371 +0000 UTC m=+23.688158000"
Jul 31 01:37:45.324184 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.324161    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mvmpt" event={"ID":"b46eb842-87fe-4021-b184-0ff856e849ad","Type":"ContainerStarted","Data":"a48661d002261f69eb28d782e2124bce4f547b0c5fc60054d032a8e7758e5b62"}
Jul 31 01:37:45.355441 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.355401    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5l5wj" podStartSLOduration=4.031850448 podStartE2EDuration="23.355387461s" totalImagesPullingTime="19.323537013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:37:45.355176958 +0000 UTC m=+23.720922586" watchObservedRunningTime="2026-07-31 01:37:45.355387461 +0000 UTC m=+23.721133088"
Jul 31 01:37:45.373908 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.373868    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tqppw" podStartSLOduration=4.084363017 podStartE2EDuration="23.373855055s" totalImagesPullingTime="19.289492038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:37:45.373525408 +0000 UTC m=+23.739271036" watchObservedRunningTime="2026-07-31 01:37:45.373855055 +0000 UTC m=+23.739600682"
Jul 31 01:37:45.386251 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.386219    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-88pgt" podStartSLOduration=4.086923907 podStartE2EDuration="23.38620587s" totalImagesPullingTime="19.299281963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:37:45.385907794 +0000 UTC m=+23.751653421" watchObservedRunningTime="2026-07-31 01:37:45.38620587 +0000 UTC m=+23.751951497"
Jul 31 01:37:45.400399 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.400359    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mvmpt" podStartSLOduration=4.105955512 podStartE2EDuration="23.40034729s" totalImagesPullingTime="19.294391778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:37:45.399841999 +0000 UTC m=+23.765587626" watchObservedRunningTime="2026-07-31 01:37:45.40034729 +0000 UTC m=+23.766092918"
Jul 31 01:37:45.414820 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.414793    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:45.508466 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.508433    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:45.557651 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:45.557615    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:46.027616 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:46.027591    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 01:37:46.059139 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:46.059015    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T01:37:46.027610204Z","UUID":"dbe4691a-20f1-4993-b44c-1b9774801d93","Handler":null,"Name":"","Endpoint":""}
Jul 31 01:37:46.064336 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:46.064317    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 01:37:46.064440 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:46.064359    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 01:37:46.146265 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:46.146191    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:46.146265 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:46.146232    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:46.146781 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:46.146306    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:46.146781 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:46.146459    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:46.328733 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:46.328695    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-685q7" event={"ID":"ada9ca69-ceee-41dd-9594-41aaf46caf56","Type":"ContainerStarted","Data":"917a666be9d5724746b09cca3d812ea3184fb65129b8d0094439bfaace164fbb"}
Jul 31 01:37:46.331040 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:46.331006    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj" event={"ID":"03ea1fd1-9975-468b-8297-41c0e36a9d00","Type":"ContainerStarted","Data":"88dc44dd47d7bc5c1b1880aa59fe8f2be85726cfb121ef255adef02170b18bf0"}
Jul 31 01:37:46.342063 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:46.342011    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-685q7" podStartSLOduration=5.067102683 podStartE2EDuration="24.341991866s" totalImagesPullingTime="19.274889183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:37:46.341775842 +0000 UTC m=+24.707521473" watchObservedRunningTime="2026-07-31 01:37:46.341991866 +0000 UTC m=+24.707737495"
Jul 31 01:37:46.459893 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:46.459799    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:46.514361 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:46.514332    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:47.145373 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:47.145341    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:47.145523 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:47.145444    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:47.312854 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:47.312825    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:47.336650 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:47.336625    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj" event={"ID":"03ea1fd1-9975-468b-8297-41c0e36a9d00","Type":"ContainerStarted","Data":"457da699ac0a4cfd4c9b21bf2debec1f72dc055ae65f6567f7d33004f144929f"}
Jul 31 01:37:47.571976 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:47.571939    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-88pgt"
Jul 31 01:37:47.572496 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:47.572471    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-88pgt"
Jul 31 01:37:47.915604 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:47.915512    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:47.972882 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:47.972853    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:48.145429 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:48.145401    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:48.145613 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:48.145401    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:48.145613 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:48.145502    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:48.145691 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:48.145647    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:48.343171 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:48.343136    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs" event={"ID":"efb771fd-993f-47ab-9e24-09976691283b","Type":"ContainerStarted","Data":"4d6acddb3a9eb910ab4830f1eb6b15b13482b3d37281e884615f06a3670517e8"}
Jul 31 01:37:48.343676 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:48.343338    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-88pgt"
Jul 31 01:37:48.344233 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:48.344068    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-88pgt"
Jul 31 01:37:48.360915 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:48.360866    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbhpj" podStartSLOduration=4.786479413 podStartE2EDuration="26.360847774s" totalImagesPullingTime="21.574368361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:37:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:37:48.360738029 +0000 UTC m=+26.726483652" watchObservedRunningTime="2026-07-31 01:37:48.360847774 +0000 UTC m=+26.726593399"
Jul 31 01:37:48.418201 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:48.418172    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:49.145530 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:49.145496    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:49.145732 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:49.145629    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:50.145672 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:50.145503    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:50.146119 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:50.145750    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:50.146157 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:50.146140    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:50.146235 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:50.146216    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:50.350213 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:50.350174    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs" event={"ID":"efb771fd-993f-47ab-9e24-09976691283b","Type":"ContainerStarted","Data":"29d71493011309a18186acaf9966741baf15d16cc9f42b3cdc3620f036e6ed80"}
Jul 31 01:37:50.350446 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:50.350418    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:50.350596 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:50.350488    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:50.352053 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:50.352030    2556 generic.go:356] "Generic (PLEG): container finished" podID="32bf4714-1aae-4a26-a0d3-cb99be970774" containerID="d0383eb61cecc69630120b7ddc953b1cb1d022ba2aa5e397da9460d2c339f202" exitCode=0
Jul 31 01:37:50.352159 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:50.352062    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq27z" event={"ID":"32bf4714-1aae-4a26-a0d3-cb99be970774","Type":"ContainerDied","Data":"d0383eb61cecc69630120b7ddc953b1cb1d022ba2aa5e397da9460d2c339f202"}
Jul 31 01:37:50.365971 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:50.365952    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:50.374302 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:50.374267    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs" podStartSLOduration=9.006526409 podStartE2EDuration="28.374254592s" totalImagesPullingTime="19.367728183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:37:50.373143979 +0000 UTC m=+28.738889764" watchObservedRunningTime="2026-07-31 01:37:50.374254592 +0000 UTC m=+28.740000220"
Jul 31 01:37:51.146073 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:51.146042    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:51.146423 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:51.146140    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:51.354301 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:51.354268    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:51.374489 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:51.374467    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:37:52.146476 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:52.146443    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:52.146961 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:52.146549    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:52.146961 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:52.146654    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:52.146961 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:52.146739    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:52.224708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:52.224679    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x94zh"]
Jul 31 01:37:52.227944 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:52.227917    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dnk4v"]
Jul 31 01:37:52.228076 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:52.228012    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:52.228140 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:52.228122    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:52.228572 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:52.228536    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vv7wq"]
Jul 31 01:37:52.358419 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:52.358391    2556 generic.go:356] "Generic (PLEG): container finished" podID="32bf4714-1aae-4a26-a0d3-cb99be970774" containerID="1bb4df3370375f5a2f3fcae35e69563819f1f0d15236155ebe59818894e8b462" exitCode=0
Jul 31 01:37:52.358569 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:52.358477    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq27z" event={"ID":"32bf4714-1aae-4a26-a0d3-cb99be970774","Type":"ContainerDied","Data":"1bb4df3370375f5a2f3fcae35e69563819f1f0d15236155ebe59818894e8b462"}
Jul 31 01:37:52.358940 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:52.358926    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:52.359063 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:52.359044    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:52.359111 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:52.359064    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:52.359211 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:52.359191    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:54.145682 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:54.145485    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:54.146189 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:54.145486    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:54.146189 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:54.145779    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:54.146189 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:54.145487    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:54.146189 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:54.145871    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:54.146189 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:54.145909    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:54.365549 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:54.365465    2556 generic.go:356] "Generic (PLEG): container finished" podID="32bf4714-1aae-4a26-a0d3-cb99be970774" containerID="0d293081fc2846921f865b70c8bd9fef2ff0333708a0c0a9ee7540da00d03180" exitCode=0
Jul 31 01:37:54.365549 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:54.365510    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq27z" event={"ID":"32bf4714-1aae-4a26-a0d3-cb99be970774","Type":"ContainerDied","Data":"0d293081fc2846921f865b70c8bd9fef2ff0333708a0c0a9ee7540da00d03180"}
Jul 31 01:37:54.868289 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:54.868252    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:54.917782 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:54.917750    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:55.852733 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:55.852700    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:55.853158 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:55.852743    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:55.853158 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:55.852875    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:55.853158 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:55.852878    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:55.853158 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:55.852936    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs podName:064895ac-87e6-4098-9f8d-13b44c5c7a10 nodeName:}" failed. No retries permitted until 2026-07-31 01:38:27.852917011 +0000 UTC m=+66.218662640 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs") pod "network-metrics-daemon-x94zh" (UID: "064895ac-87e6-4098-9f8d-13b44c5c7a10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 01:37:55.853158 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:55.852954    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret podName:32e17105-e7bb-4ad2-8ef5-77660ff9a429 nodeName:}" failed. No retries permitted until 2026-07-31 01:38:27.852945067 +0000 UTC m=+66.218690679 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret") pod "global-pull-secret-syncer-dnk4v" (UID: "32e17105-e7bb-4ad2-8ef5-77660ff9a429") : object "kube-system"/"original-pull-secret" not registered
Jul 31 01:37:55.953284 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:55.953240    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwgn\" (UniqueName: \"kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn\") pod \"network-check-target-vv7wq\" (UID: \"346810b7-b42a-44e7-8e2f-d1e7942fbe20\") " pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:55.953442 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:55.953420    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 01:37:55.953488 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:55.953444    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 01:37:55.953488 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:55.953457    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gzwgn for pod openshift-network-diagnostics/network-check-target-vv7wq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:55.953597 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:55.953523    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn podName:346810b7-b42a-44e7-8e2f-d1e7942fbe20 nodeName:}" failed. No retries permitted until 2026-07-31 01:38:27.953503314 +0000 UTC m=+66.319248938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwgn" (UniqueName: "kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn") pod "network-check-target-vv7wq" (UID: "346810b7-b42a-44e7-8e2f-d1e7942fbe20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 01:37:56.145253 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:56.145174    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:56.145253 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:56.145176    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:56.145431 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:56.145174    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:56.145431 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:56.145324    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:56.145431 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:56.145394    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:56.145552 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:56.145461    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:58.145477 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.145439    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:37:58.146221 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.145439    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:37:58.146221 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:58.145584    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dnk4v" podUID="32e17105-e7bb-4ad2-8ef5-77660ff9a429"
Jul 31 01:37:58.146221 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:58.145631    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv7wq" podUID="346810b7-b42a-44e7-8e2f-d1e7942fbe20"
Jul 31 01:37:58.146221 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.145439    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:37:58.146221 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:58.145722    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:37:58.429412 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.429379    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeReady"
Jul 31 01:37:58.429613 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.429598    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 01:37:58.472934 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.472905    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8q5rf"]
Jul 31 01:37:58.477901 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.477879    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s6rd8"]
Jul 31 01:37:58.478091 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.478026    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:58.480042 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.480014    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 01:37:58.480042 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.480038    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 01:37:58.480200 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.480097    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5txjm\""
Jul 31 01:37:58.480617 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.480602    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s6rd8"
Jul 31 01:37:58.482582 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.482347    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 01:37:58.482582 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.482364    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 01:37:58.482582 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.482390    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kcx9d\""
Jul 31 01:37:58.482582 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.482501    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 01:37:58.483750 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.483732    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s6rd8"]
Jul 31 01:37:58.486986 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.486958    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8q5rf"]
Jul 31 01:37:58.575005 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.574950    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndvqn\" (UniqueName: \"kubernetes.io/projected/90bc69db-3427-4022-83ac-b134a6898421-kube-api-access-ndvqn\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:58.575108 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.575006    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90bc69db-3427-4022-83ac-b134a6898421-config-volume\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:58.575108 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.575023    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:58.575108 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.575067    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bc69db-3427-4022-83ac-b134a6898421-tmp-dir\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:58.584242 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.584206    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:58.676384 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.676355    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxpf\" (UniqueName: \"kubernetes.io/projected/144cf673-df86-42f6-9d5a-925a3fb98dec-kube-api-access-wgxpf\") pod \"ingress-canary-s6rd8\" (UID: \"144cf673-df86-42f6-9d5a-925a3fb98dec\") " pod="openshift-ingress-canary/ingress-canary-s6rd8"
Jul 31 01:37:58.676488 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.676403    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90bc69db-3427-4022-83ac-b134a6898421-config-volume\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:58.676488 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.676422    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:58.676488 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.676461    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/144cf673-df86-42f6-9d5a-925a3fb98dec-cert\") pod \"ingress-canary-s6rd8\" (UID: \"144cf673-df86-42f6-9d5a-925a3fb98dec\") " pod="openshift-ingress-canary/ingress-canary-s6rd8"
Jul 31 01:37:58.676636 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:58.676580    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 01:37:58.676636 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.676604    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bc69db-3427-4022-83ac-b134a6898421-tmp-dir\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:58.676636 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:58.676636    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls podName:90bc69db-3427-4022-83ac-b134a6898421 nodeName:}" failed. No retries permitted until 2026-07-31 01:37:59.176617734 +0000 UTC m=+37.542363356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls") pod "dns-default-8q5rf" (UID: "90bc69db-3427-4022-83ac-b134a6898421") : secret "dns-default-metrics-tls" not found
Jul 31 01:37:58.676750 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.676704    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndvqn\" (UniqueName: \"kubernetes.io/projected/90bc69db-3427-4022-83ac-b134a6898421-kube-api-access-ndvqn\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:58.676900 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.676882    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bc69db-3427-4022-83ac-b134a6898421-tmp-dir\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:58.676992 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.676976    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90bc69db-3427-4022-83ac-b134a6898421-config-volume\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:58.684710 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.684691    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:58.688290 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.688274    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndvqn\" (UniqueName: \"kubernetes.io/projected/90bc69db-3427-4022-83ac-b134a6898421-kube-api-access-ndvqn\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:58.777910 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.777883    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxpf\" (UniqueName: \"kubernetes.io/projected/144cf673-df86-42f6-9d5a-925a3fb98dec-kube-api-access-wgxpf\") pod \"ingress-canary-s6rd8\" (UID: \"144cf673-df86-42f6-9d5a-925a3fb98dec\") " pod="openshift-ingress-canary/ingress-canary-s6rd8"
Jul 31 01:37:58.778018 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.777924    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/144cf673-df86-42f6-9d5a-925a3fb98dec-cert\") pod \"ingress-canary-s6rd8\" (UID: \"144cf673-df86-42f6-9d5a-925a3fb98dec\") " pod="openshift-ingress-canary/ingress-canary-s6rd8"
Jul 31 01:37:58.780286 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.780263    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/144cf673-df86-42f6-9d5a-925a3fb98dec-cert\") pod \"ingress-canary-s6rd8\" (UID: \"144cf673-df86-42f6-9d5a-925a3fb98dec\") " pod="openshift-ingress-canary/ingress-canary-s6rd8"
Jul 31 01:37:58.785651 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.785634    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxpf\" (UniqueName: \"kubernetes.io/projected/144cf673-df86-42f6-9d5a-925a3fb98dec-kube-api-access-wgxpf\") pod \"ingress-canary-s6rd8\" (UID: \"144cf673-df86-42f6-9d5a-925a3fb98dec\") " pod="openshift-ingress-canary/ingress-canary-s6rd8"
Jul 31 01:37:58.838045 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.837992    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s6rd8"
Jul 31 01:37:58.965233 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:58.965085    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:59.019043 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:59.019015    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s6rd8"]
Jul 31 01:37:59.022839 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:37:59.022817    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144cf673_df86_42f6_9d5a_925a3fb98dec.slice/crio-9e986995f7c005da20d6438cb6295de128706bf0f65ed310a8f8caaeae259328 WatchSource:0}: Error finding container 9e986995f7c005da20d6438cb6295de128706bf0f65ed310a8f8caaeae259328: Status 404 returned error can't find the container with id 9e986995f7c005da20d6438cb6295de128706bf0f65ed310a8f8caaeae259328
Jul 31 01:37:59.181628 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:59.181501    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:37:59.181971 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:59.181663    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 01:37:59.181971 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:37:59.181734    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls podName:90bc69db-3427-4022-83ac-b134a6898421 nodeName:}" failed. No retries permitted until 2026-07-31 01:38:00.181719832 +0000 UTC m=+38.547465438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls") pod "dns-default-8q5rf" (UID: "90bc69db-3427-4022-83ac-b134a6898421") : secret "dns-default-metrics-tls" not found
Jul 31 01:37:59.378288 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:59.378253    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s6rd8" event={"ID":"144cf673-df86-42f6-9d5a-925a3fb98dec","Type":"ContainerStarted","Data":"9e986995f7c005da20d6438cb6295de128706bf0f65ed310a8f8caaeae259328"}
Jul 31 01:37:59.380884 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:59.380855    2556 generic.go:356] "Generic (PLEG): container finished" podID="32bf4714-1aae-4a26-a0d3-cb99be970774" containerID="f20dd558ab06fd8fb016798905bdc152c465df76d9da682a65e2ec3ff33a1a10" exitCode=0
Jul 31 01:37:59.380978 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:59.380906    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq27z" event={"ID":"32bf4714-1aae-4a26-a0d3-cb99be970774","Type":"ContainerDied","Data":"f20dd558ab06fd8fb016798905bdc152c465df76d9da682a65e2ec3ff33a1a10"}
Jul 31 01:37:59.400183 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:59.400147    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:37:59.496920 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:37:59.496897    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:38:00.145869 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:00.145835    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:38:00.146061 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:00.145835    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:38:00.146061 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:00.145835    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:38:00.148663 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:00.148633    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 01:38:00.148822 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:00.148725    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 01:38:00.149762 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:00.149482    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 01:38:00.149762 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:00.149527    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 01:38:00.149762 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:00.149530    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zn99h\""
Jul 31 01:38:00.149762 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:00.149487    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gfbbx\""
Jul 31 01:38:00.188150 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:00.188124    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:38:00.188452 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:38:00.188267    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 01:38:00.188452 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:38:00.188332    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls podName:90bc69db-3427-4022-83ac-b134a6898421 nodeName:}" failed. No retries permitted until 2026-07-31 01:38:02.188313204 +0000 UTC m=+40.554058815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls") pod "dns-default-8q5rf" (UID: "90bc69db-3427-4022-83ac-b134a6898421") : secret "dns-default-metrics-tls" not found
Jul 31 01:38:00.388069 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:00.388026    2556 generic.go:356] "Generic (PLEG): container finished" podID="32bf4714-1aae-4a26-a0d3-cb99be970774" containerID="e76bbc1dd3e599f8113f2d97f50968d2fefdd9b7d317f9df9ea1d69c2d4fbcc7" exitCode=0
Jul 31 01:38:00.388210 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:00.388079    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq27z" event={"ID":"32bf4714-1aae-4a26-a0d3-cb99be970774","Type":"ContainerDied","Data":"e76bbc1dd3e599f8113f2d97f50968d2fefdd9b7d317f9df9ea1d69c2d4fbcc7"}
Jul 31 01:38:01.393541 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:01.393360    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq27z" event={"ID":"32bf4714-1aae-4a26-a0d3-cb99be970774","Type":"ContainerStarted","Data":"0af5d4e569b8d234a58996a19c8f09cefe605100667fad476a4c26635ff2c4f8"}
Jul 31 01:38:01.417223 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:01.417096    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tq27z" podStartSLOduration=9.417605558 podStartE2EDuration="39.417083472s" totalImagesPullingTime="29.999477914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:37:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:38:01.416050932 +0000 UTC m=+39.781796560" watchObservedRunningTime="2026-07-31 01:38:01.417083472 +0000 UTC m=+39.782829099"
Jul 31 01:38:01.537302 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:01.537273    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:38:01.588021 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:01.587980    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:38:02.202470 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:02.202439    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:38:02.202652 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:38:02.202534    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 01:38:02.202652 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:38:02.202597    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls podName:90bc69db-3427-4022-83ac-b134a6898421 nodeName:}" failed. No retries permitted until 2026-07-31 01:38:06.202583855 +0000 UTC m=+44.568329462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls") pod "dns-default-8q5rf" (UID: "90bc69db-3427-4022-83ac-b134a6898421") : secret "dns-default-metrics-tls" not found
Jul 31 01:38:02.325210 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:02.325181    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:38:02.376677 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:02.376640    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:38:02.397434 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:02.397411    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s6rd8" event={"ID":"144cf673-df86-42f6-9d5a-925a3fb98dec","Type":"ContainerStarted","Data":"0b4290c19003244a926c552cd5affc5cae8dc9c5025c2fdf40b2ab80defb6865"}
Jul 31 01:38:02.413309 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:02.413271    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s6rd8" podStartSLOduration=1.99519404 podStartE2EDuration="4.413257978s" totalImagesPullingTime="2.418063938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:38:02.412676052 +0000 UTC m=+40.778421679" watchObservedRunningTime="2026-07-31 01:38:02.413257978 +0000 UTC m=+40.779003606"
Jul 31 01:38:02.868098 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:02.868066    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:38:02.969000 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:02.968960    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:38:06.226615 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:06.226579    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:38:06.227117 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:38:06.226712    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 01:38:06.227117 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:38:06.226771    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls podName:90bc69db-3427-4022-83ac-b134a6898421 nodeName:}" failed. No retries permitted until 2026-07-31 01:38:14.226755147 +0000 UTC m=+52.592500753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls") pod "dns-default-8q5rf" (UID: "90bc69db-3427-4022-83ac-b134a6898421") : secret "dns-default-metrics-tls" not found
Jul 31 01:38:14.278967 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:14.278923    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:38:14.279488 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:38:14.279043    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 01:38:14.279488 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:38:14.279098    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls podName:90bc69db-3427-4022-83ac-b134a6898421 nodeName:}" failed. No retries permitted until 2026-07-31 01:38:30.279083326 +0000 UTC m=+68.644828931 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls") pod "dns-default-8q5rf" (UID: "90bc69db-3427-4022-83ac-b134a6898421") : secret "dns-default-metrics-tls" not found
Jul 31 01:38:19.372766 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:19.372707    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:38:19.420730 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:19.420667    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:38:23.371732 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:23.371700    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fsprs"
Jul 31 01:38:23.471951 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:23.471923    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:38:23.521504 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:23.521477    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:38:23.571338 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:23.571315    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:38:27.866611 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:27.866572    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:38:27.867000 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:27.866631    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:38:27.868928 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:27.868907    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 01:38:27.869027 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:27.868939    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 01:38:27.877625 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:38:27.877608    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 31 01:38:27.877698 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:38:27.877658    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs podName:064895ac-87e6-4098-9f8d-13b44c5c7a10 nodeName:}" failed. No retries permitted until 2026-07-31 01:39:31.877642754 +0000 UTC m=+130.243388359 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs") pod "network-metrics-daemon-x94zh" (UID: "064895ac-87e6-4098-9f8d-13b44c5c7a10") : secret "metrics-daemon-secret" not found
Jul 31 01:38:27.881490 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:27.881473    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32e17105-e7bb-4ad2-8ef5-77660ff9a429-original-pull-secret\") pod \"global-pull-secret-syncer-dnk4v\" (UID: \"32e17105-e7bb-4ad2-8ef5-77660ff9a429\") " pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:38:27.967183 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:27.967152    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwgn\" (UniqueName: \"kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn\") pod \"network-check-target-vv7wq\" (UID: \"346810b7-b42a-44e7-8e2f-d1e7942fbe20\") " pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:38:27.969423 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:27.969408    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 01:38:27.979923 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:27.979908    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 01:38:27.990906 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:27.990879    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzwgn\" (UniqueName: \"kubernetes.io/projected/346810b7-b42a-44e7-8e2f-d1e7942fbe20-kube-api-access-gzwgn\") pod \"network-check-target-vv7wq\" (UID: \"346810b7-b42a-44e7-8e2f-d1e7942fbe20\") " pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:38:28.060195 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:28.060172    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gfbbx\""
Jul 31 01:38:28.065091 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:28.065076    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dnk4v"
Jul 31 01:38:28.068216 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:28.068201    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:38:28.214707 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:28.214677    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dnk4v"]
Jul 31 01:38:28.220269 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:38:28.220245    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e17105_e7bb_4ad2_8ef5_77660ff9a429.slice/crio-6d0b0abda2887a8c8835fdc5236a80d907c0e52f0573061034bcf16e7ed1c3c9 WatchSource:0}: Error finding container 6d0b0abda2887a8c8835fdc5236a80d907c0e52f0573061034bcf16e7ed1c3c9: Status 404 returned error can't find the container with id 6d0b0abda2887a8c8835fdc5236a80d907c0e52f0573061034bcf16e7ed1c3c9
Jul 31 01:38:28.231794 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:28.231749    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vv7wq"]
Jul 31 01:38:28.234739 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:38:28.234718    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346810b7_b42a_44e7_8e2f_d1e7942fbe20.slice/crio-0903c6e9409bdd577a34c34e280767fbd19f7db0cfc52b7a68df762f2f12e992 WatchSource:0}: Error finding container 0903c6e9409bdd577a34c34e280767fbd19f7db0cfc52b7a68df762f2f12e992: Status 404 returned error can't find the container with id 0903c6e9409bdd577a34c34e280767fbd19f7db0cfc52b7a68df762f2f12e992
Jul 31 01:38:28.458785 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:28.458662    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vv7wq" event={"ID":"346810b7-b42a-44e7-8e2f-d1e7942fbe20","Type":"ContainerStarted","Data":"0903c6e9409bdd577a34c34e280767fbd19f7db0cfc52b7a68df762f2f12e992"}
Jul 31 01:38:28.459664 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:28.459643    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dnk4v" event={"ID":"32e17105-e7bb-4ad2-8ef5-77660ff9a429","Type":"ContainerStarted","Data":"6d0b0abda2887a8c8835fdc5236a80d907c0e52f0573061034bcf16e7ed1c3c9"}
Jul 31 01:38:30.285893 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:30.285852    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:38:30.286387 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:38:30.286012    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 01:38:30.286387 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:38:30.286075    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls podName:90bc69db-3427-4022-83ac-b134a6898421 nodeName:}" failed. No retries permitted until 2026-07-31 01:39:02.286056025 +0000 UTC m=+100.651801646 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls") pod "dns-default-8q5rf" (UID: "90bc69db-3427-4022-83ac-b134a6898421") : secret "dns-default-metrics-tls" not found
Jul 31 01:38:33.474726 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:33.474691    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vv7wq" event={"ID":"346810b7-b42a-44e7-8e2f-d1e7942fbe20","Type":"ContainerStarted","Data":"d4a09bdc883d8d17766a25534ea6c3b9f1a1b726ffc0de607668b9dd4684279e"}
Jul 31 01:38:33.475225 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:33.474784    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:38:33.476218 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:33.476193    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dnk4v" event={"ID":"32e17105-e7bb-4ad2-8ef5-77660ff9a429","Type":"ContainerStarted","Data":"4ac152b087ebd240741d64f60d819ed6ffaa6be55fc8e304784879b5a3b789cc"}
Jul 31 01:38:33.489179 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:38:33.489138    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vv7wq" podStartSLOduration=67.152443358 podStartE2EDuration="1m11.489126173s" totalImagesPullingTime="4.336682815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:38:33.488375882 +0000 UTC m=+71.854121520" watchObservedRunningTime="2026-07-31 01:38:33.489126173 +0000 UTC m=+71.854871778"
Jul 31 01:39:02.288735 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:02.288699    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:39:02.289130 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:39:02.288796    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 01:39:02.289130 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:39:02.288843    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls podName:90bc69db-3427-4022-83ac-b134a6898421 nodeName:}" failed. No retries permitted until 2026-07-31 01:40:06.288830362 +0000 UTC m=+164.654575968 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls") pod "dns-default-8q5rf" (UID: "90bc69db-3427-4022-83ac-b134a6898421") : secret "dns-default-metrics-tls" not found
Jul 31 01:39:04.481896 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:04.481870    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vv7wq"
Jul 31 01:39:04.497427 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:04.497380    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dnk4v" podStartSLOduration=98.142881635 podStartE2EDuration="1m42.497365644s" totalImagesPullingTime="4.354484009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:38:33.500374972 +0000 UTC m=+71.866120601" watchObservedRunningTime="2026-07-31 01:39:04.497365644 +0000 UTC m=+102.863111270"
Jul 31 01:39:22.569862 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.569829    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"]
Jul 31 01:39:22.571269 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.571254    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"]
Jul 31 01:39:22.571389 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.571373    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:22.572920 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.572900    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.573434 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.573419    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 31 01:39:22.573654 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.573635    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 31 01:39:22.573768 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.573664    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 31 01:39:22.573768 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.573706    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 31 01:39:22.575283 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.575260    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 31 01:39:22.575438 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.575414    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 31 01:39:22.575867 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.575847    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 31 01:39:22.576084 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.576067    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 31 01:39:22.582026 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.582002    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"]
Jul 31 01:39:22.585134 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.585114    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"]
Jul 31 01:39:22.713340 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.713308    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c2c45879-26e6-438b-bd0d-ff3ce513b92e-ca\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.713488 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.713348    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c45879-26e6-438b-bd0d-ff3ce513b92e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.713488 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.713368    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/226d0dd6-da46-4c58-aa30-9b1d825d5b99-klusterlet-config\") pod \"klusterlet-addon-workmgr-5784946ff-5b9r6\" (UID: \"226d0dd6-da46-4c58-aa30-9b1d825d5b99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:22.713488 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.713415    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2c45879-26e6-438b-bd0d-ff3ce513b92e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.713488 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.713450    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l59wc\" (UniqueName: \"kubernetes.io/projected/c2c45879-26e6-438b-bd0d-ff3ce513b92e-kube-api-access-l59wc\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.713641 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.713535    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/226d0dd6-da46-4c58-aa30-9b1d825d5b99-tmp\") pod \"klusterlet-addon-workmgr-5784946ff-5b9r6\" (UID: \"226d0dd6-da46-4c58-aa30-9b1d825d5b99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:22.713641 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.713580    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c2c45879-26e6-438b-bd0d-ff3ce513b92e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.713641 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.713608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c2c45879-26e6-438b-bd0d-ff3ce513b92e-hub\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.713641 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.713630    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlpkk\" (UniqueName: \"kubernetes.io/projected/226d0dd6-da46-4c58-aa30-9b1d825d5b99-kube-api-access-xlpkk\") pod \"klusterlet-addon-workmgr-5784946ff-5b9r6\" (UID: \"226d0dd6-da46-4c58-aa30-9b1d825d5b99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:22.814640 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.814594    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c2c45879-26e6-438b-bd0d-ff3ce513b92e-ca\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.814640 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.814651    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c45879-26e6-438b-bd0d-ff3ce513b92e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.814770 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.814670    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/226d0dd6-da46-4c58-aa30-9b1d825d5b99-klusterlet-config\") pod \"klusterlet-addon-workmgr-5784946ff-5b9r6\" (UID: \"226d0dd6-da46-4c58-aa30-9b1d825d5b99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:22.814770 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.814688    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2c45879-26e6-438b-bd0d-ff3ce513b92e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.814770 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.814704    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59wc\" (UniqueName: \"kubernetes.io/projected/c2c45879-26e6-438b-bd0d-ff3ce513b92e-kube-api-access-l59wc\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.814770 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.814742    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/226d0dd6-da46-4c58-aa30-9b1d825d5b99-tmp\") pod \"klusterlet-addon-workmgr-5784946ff-5b9r6\" (UID: \"226d0dd6-da46-4c58-aa30-9b1d825d5b99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:22.814770 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.814763    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c2c45879-26e6-438b-bd0d-ff3ce513b92e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.814987 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.814780    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c2c45879-26e6-438b-bd0d-ff3ce513b92e-hub\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.814987 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.814801    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpkk\" (UniqueName: \"kubernetes.io/projected/226d0dd6-da46-4c58-aa30-9b1d825d5b99-kube-api-access-xlpkk\") pod \"klusterlet-addon-workmgr-5784946ff-5b9r6\" (UID: \"226d0dd6-da46-4c58-aa30-9b1d825d5b99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:22.815257 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.815230    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/226d0dd6-da46-4c58-aa30-9b1d825d5b99-tmp\") pod \"klusterlet-addon-workmgr-5784946ff-5b9r6\" (UID: \"226d0dd6-da46-4c58-aa30-9b1d825d5b99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:22.815575 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.815420    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c45879-26e6-438b-bd0d-ff3ce513b92e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.817424 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.817400    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c2c45879-26e6-438b-bd0d-ff3ce513b92e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.817618 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.817596    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/226d0dd6-da46-4c58-aa30-9b1d825d5b99-klusterlet-config\") pod \"klusterlet-addon-workmgr-5784946ff-5b9r6\" (UID: \"226d0dd6-da46-4c58-aa30-9b1d825d5b99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:22.817700 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.817685    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c2c45879-26e6-438b-bd0d-ff3ce513b92e-ca\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.817803 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.817784    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2c45879-26e6-438b-bd0d-ff3ce513b92e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.818146 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.818128    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c2c45879-26e6-438b-bd0d-ff3ce513b92e-hub\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.822055 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.822006    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpkk\" (UniqueName: \"kubernetes.io/projected/226d0dd6-da46-4c58-aa30-9b1d825d5b99-kube-api-access-xlpkk\") pod \"klusterlet-addon-workmgr-5784946ff-5b9r6\" (UID: \"226d0dd6-da46-4c58-aa30-9b1d825d5b99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:22.822455 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.822433    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59wc\" (UniqueName: \"kubernetes.io/projected/c2c45879-26e6-438b-bd0d-ff3ce513b92e-kube-api-access-l59wc\") pod \"cluster-proxy-proxy-agent-7d46556b44-qb5tm\" (UID: \"c2c45879-26e6-438b-bd0d-ff3ce513b92e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:22.884390 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.884365    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:22.898667 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:22.898644    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:39:23.028174 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:23.028145    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"]
Jul 31 01:39:23.031550 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:39:23.031524    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226d0dd6_da46_4c58_aa30_9b1d825d5b99.slice/crio-2ab8eb1a349d8f1cdbdbea1ad5f2354a57ba535a22113f2559d52ad87020469c WatchSource:0}: Error finding container 2ab8eb1a349d8f1cdbdbea1ad5f2354a57ba535a22113f2559d52ad87020469c: Status 404 returned error can't find the container with id 2ab8eb1a349d8f1cdbdbea1ad5f2354a57ba535a22113f2559d52ad87020469c
Jul 31 01:39:23.049037 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:23.049011    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"]
Jul 31 01:39:23.053151 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:39:23.053129    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2c45879_26e6_438b_bd0d_ff3ce513b92e.slice/crio-9fecd5fff1f2e1f057649d7897a9f7dc7c3e55f0b87dd2cb5b5b583af1d6dc84 WatchSource:0}: Error finding container 9fecd5fff1f2e1f057649d7897a9f7dc7c3e55f0b87dd2cb5b5b583af1d6dc84: Status 404 returned error can't find the container with id 9fecd5fff1f2e1f057649d7897a9f7dc7c3e55f0b87dd2cb5b5b583af1d6dc84
Jul 31 01:39:23.591223 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:23.591186    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm" event={"ID":"c2c45879-26e6-438b-bd0d-ff3ce513b92e","Type":"ContainerStarted","Data":"9fecd5fff1f2e1f057649d7897a9f7dc7c3e55f0b87dd2cb5b5b583af1d6dc84"}
Jul 31 01:39:23.592254 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:23.592232    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6" event={"ID":"226d0dd6-da46-4c58-aa30-9b1d825d5b99","Type":"ContainerStarted","Data":"2ab8eb1a349d8f1cdbdbea1ad5f2354a57ba535a22113f2559d52ad87020469c"}
Jul 31 01:39:29.610159 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:29.610124    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6" event={"ID":"226d0dd6-da46-4c58-aa30-9b1d825d5b99","Type":"ContainerStarted","Data":"99446b563ce2db9eedbe780202fa1a88a3e8c32504cf8be83c0af2a8ae3490d7"}
Jul 31 01:39:29.610629 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:29.610312    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:29.612170 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:29.612149    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:39:29.612291 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:29.612233    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm" event={"ID":"c2c45879-26e6-438b-bd0d-ff3ce513b92e","Type":"ContainerStarted","Data":"b6b687ade8f5e25fa0242dfdffd109a5110e57f78e6de85c3e304ccf5eaca5f3"}
Jul 31 01:39:29.612291 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:29.612262    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm" event={"ID":"c2c45879-26e6-438b-bd0d-ff3ce513b92e","Type":"ContainerStarted","Data":"dd0d04292875da23ea2a36b43a44b669b64211d86195c3b7e9e5437fe624d345"}
Jul 31 01:39:29.612291 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:29.612279    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm" event={"ID":"c2c45879-26e6-438b-bd0d-ff3ce513b92e","Type":"ContainerStarted","Data":"6b7ea8cef57c100a1c8b116e3beecb9b0c98fe12bba3fbf9d42b8739a9774e24"}
Jul 31 01:39:29.625436 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:29.625387    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6" podStartSLOduration=1.903269887 podStartE2EDuration="7.625374678s" totalImagesPullingTime="5.722104791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:39:29.624651056 +0000 UTC m=+127.990396683" watchObservedRunningTime="2026-07-31 01:39:29.625374678 +0000 UTC m=+127.991120305"
Jul 31 01:39:31.886891 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:39:31.886851    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:39:31.887304 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:39:31.886953    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 31 01:39:31.887304 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:39:31.887013    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs podName:064895ac-87e6-4098-9f8d-13b44c5c7a10 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:33.886998219 +0000 UTC m=+252.252743836 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs") pod "network-metrics-daemon-x94zh" (UID: "064895ac-87e6-4098-9f8d-13b44c5c7a10") : secret "metrics-daemon-secret" not found
Jul 31 01:40:01.533675 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:40:01.533634    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8q5rf" podUID="90bc69db-3427-4022-83ac-b134a6898421"
Jul 31 01:40:01.699495 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:40:01.699468    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:40:03.171154 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:40:03.171098    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-x94zh" podUID="064895ac-87e6-4098-9f8d-13b44c5c7a10"
Jul 31 01:40:06.314014 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:40:06.313980    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:40:06.314378 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:40:06.314119    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 01:40:06.314378 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:40:06.314180    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls podName:90bc69db-3427-4022-83ac-b134a6898421 nodeName:}" failed. No retries permitted until 2026-07-31 01:42:08.314164398 +0000 UTC m=+286.679910003 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls") pod "dns-default-8q5rf" (UID: "90bc69db-3427-4022-83ac-b134a6898421") : secret "dns-default-metrics-tls" not found
Jul 31 01:40:15.146218 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:40:15.146142    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:40:41.886091 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:40:41.886057    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:06.008408 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:06.008374    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:06.105861 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:06.105836    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:07.797626 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:07.797589    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:07.897955 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:07.897926    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:09.062351 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.062293    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm" podStartSLOduration=101.35698308 podStartE2EDuration="1m47.062276812s" totalImagesPullingTime="5.705293732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:39:29.662128855 +0000 UTC m=+128.027874483" watchObservedRunningTime="2026-07-31 01:41:09.062276812 +0000 UTC m=+227.428022468"
Jul 31 01:41:09.062764 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.062447    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k4jqp"]
Jul 31 01:41:09.065401 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.065385    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.067431 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.067406    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 01:41:09.071241 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.067872    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 01:41:09.071241 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.068378    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vl56s\""
Jul 31 01:41:09.071241 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.068410    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 01:41:09.071241 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.068787    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 01:41:09.074447 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.074424    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k4jqp"]
Jul 31 01:41:09.077401 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.077381    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:09.121755 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.121724    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/926bcd5f-f88b-44fb-b4f4-acee33c2d448-crio-socket\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.121755 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.121755    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.121931 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.121778    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/926bcd5f-f88b-44fb-b4f4-acee33c2d448-data-volume\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.121931 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.121793    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.121931 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.121813    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24h28\" (UniqueName: \"kubernetes.io/projected/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-api-access-24h28\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.136352 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.136322    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:09.222775 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.222748    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24h28\" (UniqueName: \"kubernetes.io/projected/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-api-access-24h28\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.222879 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.222810    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/926bcd5f-f88b-44fb-b4f4-acee33c2d448-crio-socket\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.222879 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.222837    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.222879 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.222869    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/926bcd5f-f88b-44fb-b4f4-acee33c2d448-data-volume\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.223025 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.222892    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.223025 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.222896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/926bcd5f-f88b-44fb-b4f4-acee33c2d448-crio-socket\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.223025 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:09.222988    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 01:41:09.223158 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:09.223028    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 01:41:09.223158 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:09.223049    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls podName:926bcd5f-f88b-44fb-b4f4-acee33c2d448 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:09.723032908 +0000 UTC m=+228.088778513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k4jqp" (UID: "926bcd5f-f88b-44fb-b4f4-acee33c2d448") : secret "insights-runtime-extractor-tls" not found
Jul 31 01:41:09.223158 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:09.223077    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm podName:926bcd5f-f88b-44fb-b4f4-acee33c2d448 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:09.723063907 +0000 UTC m=+228.088809512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k4jqp" (UID: "926bcd5f-f88b-44fb-b4f4-acee33c2d448") : configmap "kube-rbac-proxy" not found
Jul 31 01:41:09.223268 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.223178    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/926bcd5f-f88b-44fb-b4f4-acee33c2d448-data-volume\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.230640 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.230621    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24h28\" (UniqueName: \"kubernetes.io/projected/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-api-access-24h28\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.727282 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.727250    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.727282 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:09.727289    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:09.727490 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:09.727350    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 01:41:09.727490 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:09.727386    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 01:41:09.727490 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:09.727399    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm podName:926bcd5f-f88b-44fb-b4f4-acee33c2d448 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:10.727385085 +0000 UTC m=+229.093130691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k4jqp" (UID: "926bcd5f-f88b-44fb-b4f4-acee33c2d448") : configmap "kube-rbac-proxy" not found
Jul 31 01:41:09.727490 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:09.727440    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls podName:926bcd5f-f88b-44fb-b4f4-acee33c2d448 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:10.727427704 +0000 UTC m=+229.093173309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k4jqp" (UID: "926bcd5f-f88b-44fb-b4f4-acee33c2d448") : secret "insights-runtime-extractor-tls" not found
Jul 31 01:41:10.088273 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:10.088244    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:10.153760 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:10.153731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:10.204053 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:10.204024    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:10.543939 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:10.543860    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:10.644488 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:10.644466    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:10.736697 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:10.736668    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:10.736809 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:10.736703    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:10.736809 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:10.736772    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 01:41:10.736872 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:10.736809    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 01:41:10.736872 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:10.736822    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm podName:926bcd5f-f88b-44fb-b4f4-acee33c2d448 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:12.73680767 +0000 UTC m=+231.102553276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k4jqp" (UID: "926bcd5f-f88b-44fb-b4f4-acee33c2d448") : configmap "kube-rbac-proxy" not found
Jul 31 01:41:10.736872 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:10.736870    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls podName:926bcd5f-f88b-44fb-b4f4-acee33c2d448 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:12.736854677 +0000 UTC m=+231.102600299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k4jqp" (UID: "926bcd5f-f88b-44fb-b4f4-acee33c2d448") : secret "insights-runtime-extractor-tls" not found
Jul 31 01:41:10.974027 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:10.973998    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:11.022842 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:11.022814    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:11.219150 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:11.219124    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tqppw_8525e366-5355-4406-8e20-88f336bf1e44/dns-node-resolver/0.log"
Jul 31 01:41:11.258210 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:11.258146    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:11.308446 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:11.308419    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:12.012535 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:12.012508    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lprjl_d5c6c5a5-2a0c-48e2-b968-93837c492672/node-ca/0.log"
Jul 31 01:41:12.750515 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:12.750478    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:12.750515 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:12.750520    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:12.750953 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:12.750606    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 01:41:12.750953 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:12.750642    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 01:41:12.750953 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:12.750657    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm podName:926bcd5f-f88b-44fb-b4f4-acee33c2d448 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:16.750643044 +0000 UTC m=+235.116388649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k4jqp" (UID: "926bcd5f-f88b-44fb-b4f4-acee33c2d448") : configmap "kube-rbac-proxy" not found
Jul 31 01:41:12.750953 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:12.750690    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls podName:926bcd5f-f88b-44fb-b4f4-acee33c2d448 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:16.750678693 +0000 UTC m=+235.116424298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k4jqp" (UID: "926bcd5f-f88b-44fb-b4f4-acee33c2d448") : secret "insights-runtime-extractor-tls" not found
Jul 31 01:41:12.815201 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:12.815177    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:12.914148 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:12.914119    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:13.013804 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:13.013745    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s6rd8_144cf673-df86-42f6-9d5a-925a3fb98dec/serve-healthcheck-canary/0.log"
Jul 31 01:41:13.822227 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:13.822191    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:14.897848 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:14.897821    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:15.032735 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:15.032703    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:15.087320 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:15.087290    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:15.144320 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:15.144299    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:15.254354 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:15.254285    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:15.447511 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:15.447478    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:15.589631 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:15.589601    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:16.093700 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:16.093666    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:16.589271 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:16.589243    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:16.776584 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:16.776530    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:16.776758 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:16.776596    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:16.776758 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:16.776670    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 01:41:16.776758 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:16.776697    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 01:41:16.776758 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:16.776730    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls podName:926bcd5f-f88b-44fb-b4f4-acee33c2d448 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:24.776715307 +0000 UTC m=+243.142460912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k4jqp" (UID: "926bcd5f-f88b-44fb-b4f4-acee33c2d448") : secret "insights-runtime-extractor-tls" not found
Jul 31 01:41:16.776758 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:16.776749    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm podName:926bcd5f-f88b-44fb-b4f4-acee33c2d448 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:24.77673538 +0000 UTC m=+243.142481009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k4jqp" (UID: "926bcd5f-f88b-44fb-b4f4-acee33c2d448") : configmap "kube-rbac-proxy" not found
Jul 31 01:41:16.994109 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:16.994030    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:17.186468 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:17.186440    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:17.387486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:17.387460    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:17.786398 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:17.786320    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:18.586339 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:18.586305    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:18.687954 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:18.687926    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:18.787314 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:18.787287    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:18.844116 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:18.844054    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:18.983968 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:18.983940    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:19.188222 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:19.188149    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:19.987350 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:19.987321    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:20.389363 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:20.389334    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:20.443381 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:20.443350    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:20.786872 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:20.786791    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:20.985319 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:20.985290    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:21.368928 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:21.368901    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:21.417811 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:21.417784    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:21.787134 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:21.787054    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:22.329873 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:22.329842    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:22.430049 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:22.430017    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:22.798888 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:22.798810    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:22.983804 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:22.983768    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:23.387710 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:23.387680    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:24.184454 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:24.184421    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:24.387930 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:24.387901    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:24.785820 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:24.785786    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:24.837104 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:24.837075    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:24.837227 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:24.837111    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:24.837227 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:24.837204    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 01:41:24.837305 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:24.837277    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls podName:926bcd5f-f88b-44fb-b4f4-acee33c2d448 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:40.837263748 +0000 UTC m=+259.203009353 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k4jqp" (UID: "926bcd5f-f88b-44fb-b4f4-acee33c2d448") : secret "insights-runtime-extractor-tls" not found
Jul 31 01:41:24.837527 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:24.837512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/926bcd5f-f88b-44fb-b4f4-acee33c2d448-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:24.985956 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:24.985928    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:25.390575 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:25.390533    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:25.587116 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:25.587086    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:28.052853 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:28.052818    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:28.157680 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:28.157649    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:28.356075 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:28.356050    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:28.534336 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:28.534307    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:28.585107 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:28.585084    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:28.931928 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:28.931900    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:28.942090 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:28.942063    2556 generic.go:356] "Generic (PLEG): container finished" podID="226d0dd6-da46-4c58-aa30-9b1d825d5b99" containerID="99446b563ce2db9eedbe780202fa1a88a3e8c32504cf8be83c0af2a8ae3490d7" exitCode=1
Jul 31 01:41:28.942196 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:28.942112    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6" event={"ID":"226d0dd6-da46-4c58-aa30-9b1d825d5b99","Type":"ContainerDied","Data":"99446b563ce2db9eedbe780202fa1a88a3e8c32504cf8be83c0af2a8ae3490d7"}
Jul 31 01:41:28.942419 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:28.942392    2556 scope.go:117] "RemoveContainer" containerID="99446b563ce2db9eedbe780202fa1a88a3e8c32504cf8be83c0af2a8ae3490d7"
Jul 31 01:41:28.989330 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:28.989308    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:29.331379 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:29.331352    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:29.380398 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:29.380373    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:29.610363 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:29.610294    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:41:29.744316 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:29.744287    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:29.944143 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:29.944062    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:29.946346 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:29.946312    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6" event={"ID":"226d0dd6-da46-4c58-aa30-9b1d825d5b99","Type":"ContainerStarted","Data":"0b81b88f7096550293e1ba4b2986ba7c5c2e117817080c76dac7675ba79830df"}
Jul 31 01:41:29.946521 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:29.946507    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:41:29.947111 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:29.947097    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5784946ff-5b9r6"
Jul 31 01:41:30.131739 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:30.131707    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:30.759265 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:30.759232    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:30.860090 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:30.860058    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:31.801410 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:31.801381    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-w6ktx"]
Jul 31 01:41:31.804601 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:31.804582    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-w6ktx"
Jul 31 01:41:31.806771 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:31.806752    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rk76m\""
Jul 31 01:41:31.807018 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:31.806931    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 31 01:41:31.807106 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:31.807069    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 31 01:41:31.814334 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:31.814313    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-w6ktx"]
Jul 31 01:41:31.889553 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:31.889522    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554ss\" (UniqueName: \"kubernetes.io/projected/b1ea40d4-e2d3-46b7-b4a6-9d54bfefdb0f-kube-api-access-554ss\") pod \"downloads-77c974ffcf-w6ktx\" (UID: \"b1ea40d4-e2d3-46b7-b4a6-9d54bfefdb0f\") " pod="openshift-console/downloads-77c974ffcf-w6ktx"
Jul 31 01:41:31.901751 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:31.901722    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:31.990605 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:31.990577    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554ss\" (UniqueName: \"kubernetes.io/projected/b1ea40d4-e2d3-46b7-b4a6-9d54bfefdb0f-kube-api-access-554ss\") pod \"downloads-77c974ffcf-w6ktx\" (UID: \"b1ea40d4-e2d3-46b7-b4a6-9d54bfefdb0f\") " pod="openshift-console/downloads-77c974ffcf-w6ktx"
Jul 31 01:41:31.999776 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:31.999748    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-554ss\" (UniqueName: \"kubernetes.io/projected/b1ea40d4-e2d3-46b7-b4a6-9d54bfefdb0f-kube-api-access-554ss\") pod \"downloads-77c974ffcf-w6ktx\" (UID: \"b1ea40d4-e2d3-46b7-b4a6-9d54bfefdb0f\") " pod="openshift-console/downloads-77c974ffcf-w6ktx"
Jul 31 01:41:32.005901 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:32.005879    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:32.112647 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:32.112623    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-w6ktx"
Jul 31 01:41:32.232461 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:32.232432    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-w6ktx"]
Jul 31 01:41:32.236355 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:41:32.236326    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ea40d4_e2d3_46b7_b4a6_9d54bfefdb0f.slice/crio-146b966d8679c043ee27bc266770d04e6aa6c8d8d5c2c41ef3fd99364cb3c28f WatchSource:0}: Error finding container 146b966d8679c043ee27bc266770d04e6aa6c8d8d5c2c41ef3fd99364cb3c28f: Status 404 returned error can't find the container with id 146b966d8679c043ee27bc266770d04e6aa6c8d8d5c2c41ef3fd99364cb3c28f
Jul 31 01:41:32.875328 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:32.875283    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:32.925183 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:32.925149    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:32.956087 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:32.956002    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-w6ktx" event={"ID":"b1ea40d4-e2d3-46b7-b4a6-9d54bfefdb0f","Type":"ContainerStarted","Data":"146b966d8679c043ee27bc266770d04e6aa6c8d8d5c2c41ef3fd99364cb3c28f"}
Jul 31 01:41:33.896966 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:33.896924    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:33.906077 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:33.906045    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:41:33.908662 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:33.908644    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064895ac-87e6-4098-9f8d-13b44c5c7a10-metrics-certs\") pod \"network-metrics-daemon-x94zh\" (UID: \"064895ac-87e6-4098-9f8d-13b44c5c7a10\") " pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:41:33.997240 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:33.997207    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:34.049311 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:34.049274    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zn99h\""
Jul 31 01:41:34.057025 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:34.056996    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:34.057840 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:34.057823    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x94zh"
Jul 31 01:41:34.195127 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:34.195098    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x94zh"]
Jul 31 01:41:34.197788 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:41:34.197765    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064895ac_87e6_4098_9f8d_13b44c5c7a10.slice/crio-4ed0da3b97e7da70be56df111b8e4e75903420c2487f438661c7c278f885cbc0 WatchSource:0}: Error finding container 4ed0da3b97e7da70be56df111b8e4e75903420c2487f438661c7c278f885cbc0: Status 404 returned error can't find the container with id 4ed0da3b97e7da70be56df111b8e4e75903420c2487f438661c7c278f885cbc0
Jul 31 01:41:34.968538 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:34.968495    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x94zh" event={"ID":"064895ac-87e6-4098-9f8d-13b44c5c7a10","Type":"ContainerStarted","Data":"4ed0da3b97e7da70be56df111b8e4e75903420c2487f438661c7c278f885cbc0"}
Jul 31 01:41:35.974617 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:35.974585    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x94zh" event={"ID":"064895ac-87e6-4098-9f8d-13b44c5c7a10","Type":"ContainerStarted","Data":"e784d4bff311afe57a772f89702ecfa44405f35e51e274746533dba90f217cc9"}
Jul 31 01:41:35.975023 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:35.974640    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x94zh" event={"ID":"064895ac-87e6-4098-9f8d-13b44c5c7a10","Type":"ContainerStarted","Data":"e1303e3a3f7eec343c1a42f67132e91882969297beafdd157b63c3820bdfb415"}
Jul 31 01:41:35.990032 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:35.989983    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x94zh" podStartSLOduration=253.086970924 podStartE2EDuration="4m13.989962661s" totalImagesPullingTime="902.991737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:41:35.988971511 +0000 UTC m=+254.354717138" watchObservedRunningTime="2026-07-31 01:41:35.989962661 +0000 UTC m=+254.355708288"
Jul 31 01:41:36.079875 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:36.079841    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:36.150178 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:36.150151    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:38.970503 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:38.970461    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:39.018820 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:39.018789    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:40.861715 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:40.861676    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:40.864389 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:40.864360    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/926bcd5f-f88b-44fb-b4f4-acee33c2d448-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4jqp\" (UID: \"926bcd5f-f88b-44fb-b4f4-acee33c2d448\") " pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:40.880454 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:40.880426    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vl56s\""
Jul 31 01:41:40.889382 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:40.889362    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k4jqp"
Jul 31 01:41:41.027538 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.027497    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k4jqp"]
Jul 31 01:41:41.030576 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:41:41.030530    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926bcd5f_f88b_44fb_b4f4_acee33c2d448.slice/crio-1a70383e9eb3ccced00edf64de05a948cfa7e428c7f026c4dd7f669186f10946 WatchSource:0}: Error finding container 1a70383e9eb3ccced00edf64de05a948cfa7e428c7f026c4dd7f669186f10946: Status 404 returned error can't find the container with id 1a70383e9eb3ccced00edf64de05a948cfa7e428c7f026c4dd7f669186f10946
Jul 31 01:41:41.146640 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.146568    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:41.246496 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.246462    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:41.350362 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.350320    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv"]
Jul 31 01:41:41.353979 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.353949    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv"
Jul 31 01:41:41.356378 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.356147    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 31 01:41:41.356378 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.356188    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kcngf\""
Jul 31 01:41:41.360186 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.360163    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv"]
Jul 31 01:41:41.363930 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.363877    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:41.466910 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.466827    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b433e829-5435-43f5-912b-1b6a65e29be4-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-lgpmv\" (UID: \"b433e829-5435-43f5-912b-1b6a65e29be4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv"
Jul 31 01:41:41.545352 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.545324    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:41.567502 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.567472    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b433e829-5435-43f5-912b-1b6a65e29be4-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-lgpmv\" (UID: \"b433e829-5435-43f5-912b-1b6a65e29be4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv"
Jul 31 01:41:41.570196 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.570171    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b433e829-5435-43f5-912b-1b6a65e29be4-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-lgpmv\" (UID: \"b433e829-5435-43f5-912b-1b6a65e29be4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv"
Jul 31 01:41:41.665984 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.665945    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv"
Jul 31 01:41:41.812403 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:41.812365    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv"]
Jul 31 01:41:41.816164 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:41:41.816133    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb433e829_5435_43f5_912b_1b6a65e29be4.slice/crio-f23604189002758034a6e3e8f96730608e16858a5cbaee23fdbf2772163d629b WatchSource:0}: Error finding container f23604189002758034a6e3e8f96730608e16858a5cbaee23fdbf2772163d629b: Status 404 returned error can't find the container with id f23604189002758034a6e3e8f96730608e16858a5cbaee23fdbf2772163d629b
Jul 31 01:41:42.000229 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:42.000152    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv" event={"ID":"b433e829-5435-43f5-912b-1b6a65e29be4","Type":"ContainerStarted","Data":"f23604189002758034a6e3e8f96730608e16858a5cbaee23fdbf2772163d629b"}
Jul 31 01:41:42.002194 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:42.002166    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4jqp" event={"ID":"926bcd5f-f88b-44fb-b4f4-acee33c2d448","Type":"ContainerStarted","Data":"a60cd0db515861dfe7b500c555a294708cef08449a1445aa46ebf7bc203973dc"}
Jul 31 01:41:42.002310 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:42.002219    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4jqp" event={"ID":"926bcd5f-f88b-44fb-b4f4-acee33c2d448","Type":"ContainerStarted","Data":"3ecebff2db73fa85c1ba16073d48a6246789480e07d10605bc5389ff9b4a0dc0"}
Jul 31 01:41:42.002310 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:42.002243    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4jqp" event={"ID":"926bcd5f-f88b-44fb-b4f4-acee33c2d448","Type":"ContainerStarted","Data":"1a70383e9eb3ccced00edf64de05a948cfa7e428c7f026c4dd7f669186f10946"}
Jul 31 01:41:42.364945 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:42.364912    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:42.418024 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:42.417988    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:45.969871 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:45.969838    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:46.036422 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:46.036389    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:47.966163 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:47.966123    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:48.024551 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:48.024520    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:49.030607 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:49.030508    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-w6ktx" event={"ID":"b1ea40d4-e2d3-46b7-b4a6-9d54bfefdb0f","Type":"ContainerStarted","Data":"31783a446c28fbc99d335edc0f7e912392fccba81ee4a25c3070e2c7e3a1d827"}
Jul 31 01:41:49.032192 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:49.032167    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-w6ktx"
Jul 31 01:41:49.038210 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:49.038172    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-w6ktx"
Jul 31 01:41:49.049049 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:49.048996    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-w6ktx" podStartSLOduration=1.56955959 podStartE2EDuration="18.048976461s" totalImagesPullingTime="16.479416871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:41:49.047052816 +0000 UTC m=+267.412798444" watchObservedRunningTime="2026-07-31 01:41:49.048976461 +0000 UTC m=+267.414722086"
Jul 31 01:41:51.041938 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:51.041873    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4jqp" event={"ID":"926bcd5f-f88b-44fb-b4f4-acee33c2d448","Type":"ContainerStarted","Data":"692a34ca96a50f932e0d211dc63a708b495d2b6e35306c94b96803f3c84cc3b0"}
Jul 31 01:41:51.043642 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:51.043612    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv" event={"ID":"b433e829-5435-43f5-912b-1b6a65e29be4","Type":"ContainerStarted","Data":"9f5e9f4143821a8f233ce8361e9df008ed018f61525a8067ba52ad8e55234f49"}
Jul 31 01:41:51.060000 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:51.059945    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k4jqp" podStartSLOduration=33.264050832 podStartE2EDuration="42.059928859s" totalImagesPullingTime="8.795878027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:41:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:41:51.059259358 +0000 UTC m=+269.425004987" watchObservedRunningTime="2026-07-31 01:41:51.059928859 +0000 UTC m=+269.425674486"
Jul 31 01:41:51.073592 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:51.073520    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv" podStartSLOduration=1.9201039880000002 podStartE2EDuration="10.073501497s" totalImagesPullingTime="8.153397509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:41:51.072488424 +0000 UTC m=+269.438234053" watchObservedRunningTime="2026-07-31 01:41:51.073501497 +0000 UTC m=+269.439247125"
Jul 31 01:41:52.047997 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:52.047963    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv"
Jul 31 01:41:52.052095 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:52.052070    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-lgpmv"
Jul 31 01:41:52.366601 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:52.366542    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:52.433023 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:52.432988    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:52.502052 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:52.502015    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:53.110709 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:53.110672    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:53.161658 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:53.161628    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:54.260534 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:54.260492    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:56.515606 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.515573    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:56.663902 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.663865    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:56.741729 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.741697    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:56.773050 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.772976    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dxbfb"]
Jul 31 01:41:56.805048 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.805020    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:56.807285 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.807263    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 01:41:56.807465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.807362    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-94jhd\""
Jul 31 01:41:56.807677 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.807641    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 01:41:56.808457 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.808277    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 01:41:56.808457 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.808300    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 01:41:56.808457 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.808329    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 01:41:56.808457 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.808302    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 01:41:56.941030 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.940996    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:56.985184 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.985149    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b86f0a5-bacf-430e-b61b-21769d948195-root\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:56.985345 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.985197    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-textfile\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:56.985345 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.985292    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-wtmp\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:56.985345 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.985324    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrwc9\" (UniqueName: \"kubernetes.io/projected/2b86f0a5-bacf-430e-b61b-21769d948195-kube-api-access-xrwc9\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:56.985494 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.985345    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b86f0a5-bacf-430e-b61b-21769d948195-sys\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:56.985494 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.985446    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b86f0a5-bacf-430e-b61b-21769d948195-metrics-client-ca\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:56.985494 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.985473    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-accelerators-collector-config\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:56.985494 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.985493    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:56.985676 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:56.985515    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-tls\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.041498 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.041415    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:57.086690 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.086661    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-wtmp\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.086847 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.086699    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrwc9\" (UniqueName: \"kubernetes.io/projected/2b86f0a5-bacf-430e-b61b-21769d948195-kube-api-access-xrwc9\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.086847 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.086721    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b86f0a5-bacf-430e-b61b-21769d948195-sys\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.086847 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.086799    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b86f0a5-bacf-430e-b61b-21769d948195-sys\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.086996 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.086854    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-wtmp\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.086996 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.086912    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b86f0a5-bacf-430e-b61b-21769d948195-metrics-client-ca\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.086996 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.086948    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-accelerators-collector-config\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.086996 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.086983    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.087175 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.087007    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-tls\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.087175 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.087054    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b86f0a5-bacf-430e-b61b-21769d948195-root\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.087175 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.087102    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b86f0a5-bacf-430e-b61b-21769d948195-root\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.087175 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.087132    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-textfile\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.087175 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:57.087157    2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 31 01:41:57.087406 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:41:57.087221    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-tls podName:2b86f0a5-bacf-430e-b61b-21769d948195 nodeName:}" failed. No retries permitted until 2026-07-31 01:41:57.587201004 +0000 UTC m=+275.952946610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-tls") pod "node-exporter-dxbfb" (UID: "2b86f0a5-bacf-430e-b61b-21769d948195") : secret "node-exporter-tls" not found
Jul 31 01:41:57.087468 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.087417    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-textfile\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.087673 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.087646    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-accelerators-collector-config\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.087929 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.087914    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b86f0a5-bacf-430e-b61b-21769d948195-metrics-client-ca\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.089883 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.089860    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.095359 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.095335    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrwc9\" (UniqueName: \"kubernetes.io/projected/2b86f0a5-bacf-430e-b61b-21769d948195-kube-api-access-xrwc9\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.282486 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.282452    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:57.382925 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.382896    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:57.483375 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.483342    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:57.590028 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.589994    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-tls\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.592856 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.592831    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b86f0a5-bacf-430e-b61b-21769d948195-node-exporter-tls\") pod \"node-exporter-dxbfb\" (UID: \"2b86f0a5-bacf-430e-b61b-21769d948195\") " pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.715477 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.715406    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dxbfb"
Jul 31 01:41:57.722326 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.722301    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:57.753634 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:41:57.753599    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b86f0a5_bacf_430e_b61b_21769d948195.slice/crio-ee1f77813f3df3c3a02a5ec03ee31fd678315b8423a4b20349008966c2a4ebd2 WatchSource:0}: Error finding container ee1f77813f3df3c3a02a5ec03ee31fd678315b8423a4b20349008966c2a4ebd2: Status 404 returned error can't find the container with id ee1f77813f3df3c3a02a5ec03ee31fd678315b8423a4b20349008966c2a4ebd2
Jul 31 01:41:57.798860 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:57.798833    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:58.068669 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:58.068628    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dxbfb" event={"ID":"2b86f0a5-bacf-430e-b61b-21769d948195","Type":"ContainerStarted","Data":"ee1f77813f3df3c3a02a5ec03ee31fd678315b8423a4b20349008966c2a4ebd2"}
Jul 31 01:41:58.916976 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:58.916939    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:59.705583 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:59.705527    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:59.768622 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:59.768591    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:59.836757 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:59.836676    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:41:59.959944 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:41:59.959905    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:00.076817 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:00.076786    2556 generic.go:356] "Generic (PLEG): container finished" podID="2b86f0a5-bacf-430e-b61b-21769d948195" containerID="dacd1edc81a8c9a7a7418b1f7620c757fde4cd7d41e275f7f77ded1733bc0e09" exitCode=0
Jul 31 01:42:00.076951 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:00.076820    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dxbfb" event={"ID":"2b86f0a5-bacf-430e-b61b-21769d948195","Type":"ContainerDied","Data":"dacd1edc81a8c9a7a7418b1f7620c757fde4cd7d41e275f7f77ded1733bc0e09"}
Jul 31 01:42:00.162008 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:00.161975    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:01.082324 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:01.082295    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dxbfb" event={"ID":"2b86f0a5-bacf-430e-b61b-21769d948195","Type":"ContainerStarted","Data":"aaafadd84b8ff7692524ac13013c49a8ee5abea915cda15e0168aa20ecab32a4"}
Jul 31 01:42:01.082798 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:01.082353    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dxbfb" event={"ID":"2b86f0a5-bacf-430e-b61b-21769d948195","Type":"ContainerStarted","Data":"5a9ab1293a4121f4fe469fe3c3471b78fef39dd406b8c9cdfc94af3d40d7dca4"}
Jul 31 01:42:01.100338 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:01.100299    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dxbfb" podStartSLOduration=3.415445368 podStartE2EDuration="5.10028576s" totalImagesPullingTime="1.581215337s" totalInitContainerRuntime="103.625055ms" isStatefulPod=true podCreationTimestamp="2026-07-31 01:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:42:01.099448353 +0000 UTC m=+279.465193981" watchObservedRunningTime="2026-07-31 01:42:01.10028576 +0000 UTC m=+279.466031385"
Jul 31 01:42:01.540851 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:01.540775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:01.616255 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:01.616227    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:01.812131 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:01.812106    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:02.939632 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.939597    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 01:42:02.944148 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.944124    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:02.962852 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.962832    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 01:42:02.963016 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.962963    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:02.965245 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.965220    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 31 01:42:02.965386 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.965369    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eq8fggc56ml5t\""
Jul 31 01:42:02.965463 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.965434    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 31 01:42:02.965634 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.965620    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 31 01:42:02.965709 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.965651    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 31 01:42:02.965709 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.965620    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 31 01:42:02.965808 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.965627    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 31 01:42:02.966054 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.966023    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 31 01:42:02.966156 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.966049    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 31 01:42:02.966156 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.966082    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 31 01:42:02.966156 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.966025    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 31 01:42:02.966156 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.966112    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 31 01:42:02.966360 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.966236    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 31 01:42:02.966360 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.966263    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-62brm\""
Jul 31 01:42:02.966642 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.966625    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 31 01:42:02.969142 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.969124    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 31 01:42:02.970499 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:02.970479    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 31 01:42:03.028021 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.027997    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028137 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028030    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028137 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028046    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028137 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028076    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028260 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028150    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028260 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028178    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028260 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028197    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2baf2933-1533-4bc7-8243-c1b7e6a011ad-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028260 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028219    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028260 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028466 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028288    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028466 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028311    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-web-config\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028466 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028348    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2baf2933-1533-4bc7-8243-c1b7e6a011ad-config-out\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028466 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028380    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028466 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028409    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kxfp\" (UniqueName: \"kubernetes.io/projected/2baf2933-1533-4bc7-8243-c1b7e6a011ad-kube-api-access-9kxfp\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028466 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028435    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028723 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028471    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-config\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028723 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028495    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028723 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028547    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028723 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028584    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.028723 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.028602    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129392 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129367    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-config\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129392 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129394    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129598 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129412    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129598 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129427    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129598 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129442    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129598 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129459    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129598 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129480    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129598 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129501    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129598 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129526    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129598 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129593    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129898 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129614    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129898 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129640    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2baf2933-1533-4bc7-8243-c1b7e6a011ad-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129898 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129667    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129898 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129684    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129898 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129723    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129898 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129748    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-web-config\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129898 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129775    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2baf2933-1533-4bc7-8243-c1b7e6a011ad-config-out\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129898 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129797    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129898 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129817    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kxfp\" (UniqueName: \"kubernetes.io/projected/2baf2933-1533-4bc7-8243-c1b7e6a011ad-kube-api-access-9kxfp\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129898 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129845    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.129898 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.129857    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.130683 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.130661    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.130861 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.130672    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.131267 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.131248    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.132493 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.132320    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.134027 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.132842    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.134027 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.133682    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.134027 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.133734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2baf2933-1533-4bc7-8243-c1b7e6a011ad-config-out\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.134027 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.133926    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.134027 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.133961    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.134644 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.134029    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.134644 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.134257    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baf2933-1533-4bc7-8243-c1b7e6a011ad-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.135031 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.135007    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-config\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.135108 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.135051    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.135200 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.135171    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2baf2933-1533-4bc7-8243-c1b7e6a011ad-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.135804 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.135779    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.135804 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.135792    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-web-config\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.135973 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.135951    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.136216 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.136198    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2baf2933-1533-4bc7-8243-c1b7e6a011ad-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.141126 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.141107    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kxfp\" (UniqueName: \"kubernetes.io/projected/2baf2933-1533-4bc7-8243-c1b7e6a011ad-kube-api-access-9kxfp\") pod \"prometheus-k8s-0\" (UID: \"2baf2933-1533-4bc7-8243-c1b7e6a011ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.272270 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.272196    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:03.422043 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:03.422017    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 01:42:03.423579 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:42:03.423529    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2baf2933_1533_4bc7_8243_c1b7e6a011ad.slice/crio-66a59ffa91b45f6482574cbe49c43832be88ba682f12f183e66f298cd6fb4d8c WatchSource:0}: Error finding container 66a59ffa91b45f6482574cbe49c43832be88ba682f12f183e66f298cd6fb4d8c: Status 404 returned error can't find the container with id 66a59ffa91b45f6482574cbe49c43832be88ba682f12f183e66f298cd6fb4d8c
Jul 31 01:42:04.092942 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:04.092904    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2baf2933-1533-4bc7-8243-c1b7e6a011ad","Type":"ContainerStarted","Data":"66a59ffa91b45f6482574cbe49c43832be88ba682f12f183e66f298cd6fb4d8c"}
Jul 31 01:42:04.303419 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:04.303386    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:04.700462 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:42:04.700418    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8q5rf" podUID="90bc69db-3427-4022-83ac-b134a6898421"
Jul 31 01:42:05.096579 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:05.096542    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:42:06.101804 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:06.101766    2556 generic.go:356] "Generic (PLEG): container finished" podID="2baf2933-1533-4bc7-8243-c1b7e6a011ad" containerID="ed92148857e46ad2eb610298dd95da275f958a8208b4355ee661c6f846c43c29" exitCode=0
Jul 31 01:42:06.102256 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:06.101830    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2baf2933-1533-4bc7-8243-c1b7e6a011ad","Type":"ContainerDied","Data":"ed92148857e46ad2eb610298dd95da275f958a8208b4355ee661c6f846c43c29"}
Jul 31 01:42:06.890158 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:06.890125    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:06.947353 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:06.947317    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:08.147543 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:08.147514    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:08.196956 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:08.196876    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:08.371128 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:08.371086    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:42:08.373788 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:08.373756    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90bc69db-3427-4022-83ac-b134a6898421-metrics-tls\") pod \"dns-default-8q5rf\" (UID: \"90bc69db-3427-4022-83ac-b134a6898421\") " pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:42:08.400017 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:08.399943    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5txjm\""
Jul 31 01:42:08.408881 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:08.408853    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:42:09.410326 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:09.410294    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:09.431453 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:09.431420    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8q5rf"]
Jul 31 01:42:09.434181 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:42:09.434157    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bc69db_3427_4022_83ac_b134a6898421.slice/crio-b4fc16e1a2e5185a6c45204f6345ec232796eab2d04b4a1d4325fbcc80e4b966 WatchSource:0}: Error finding container b4fc16e1a2e5185a6c45204f6345ec232796eab2d04b4a1d4325fbcc80e4b966: Status 404 returned error can't find the container with id b4fc16e1a2e5185a6c45204f6345ec232796eab2d04b4a1d4325fbcc80e4b966
Jul 31 01:42:09.498114 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:09.498086    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:10.120536 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:10.120487    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2baf2933-1533-4bc7-8243-c1b7e6a011ad","Type":"ContainerStarted","Data":"24e234fb7dc2469e852315621c112d28742c0f2fdfd5d2e703e2703b4e8506f0"}
Jul 31 01:42:10.120727 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:10.120638    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2baf2933-1533-4bc7-8243-c1b7e6a011ad","Type":"ContainerStarted","Data":"3d347dd74af0654e5e98c3ae75c1c1adc42c1eb8636889f064622874eda5d508"}
Jul 31 01:42:10.121662 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:10.121633    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8q5rf" event={"ID":"90bc69db-3427-4022-83ac-b134a6898421","Type":"ContainerStarted","Data":"b4fc16e1a2e5185a6c45204f6345ec232796eab2d04b4a1d4325fbcc80e4b966"}
Jul 31 01:42:12.163898 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:12.163865    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:12.266032 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:12.266005    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:12.367002 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:12.366971    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:12.899840 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:12.899805    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm" podUID="c2c45879-26e6-438b-bd0d-ff3ce513b92e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 31 01:42:13.134331 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:13.134293    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2baf2933-1533-4bc7-8243-c1b7e6a011ad","Type":"ContainerStarted","Data":"2f5f9be688853594a9fb4c74d6c15b2f7a1b6cabf48087a9b8b96c5067dea6b4"}
Jul 31 01:42:13.134536 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:13.134405    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2baf2933-1533-4bc7-8243-c1b7e6a011ad","Type":"ContainerStarted","Data":"92f5dec216c686ae873e6111aaa3057c6031ab5aabff0c39759f18558858937c"}
Jul 31 01:42:13.134536 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:13.134445    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2baf2933-1533-4bc7-8243-c1b7e6a011ad","Type":"ContainerStarted","Data":"e2c2cae5c67a5a65adc0be7e1a0c48dfa1d95fa0637e1a9730f11c3fc630c496"}
Jul 31 01:42:13.134536 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:13.134478    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2baf2933-1533-4bc7-8243-c1b7e6a011ad","Type":"ContainerStarted","Data":"fa30b1e63d4cae92f4ff8d6ddc93d5928b9bc6d09532f829ebe3edd79122624d"}
Jul 31 01:42:13.136043 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:13.136022    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8q5rf" event={"ID":"90bc69db-3427-4022-83ac-b134a6898421","Type":"ContainerStarted","Data":"98d699569daa4c554ba4c109375d66dcec902b9638d52f2ce6dcdff55f79671a"}
Jul 31 01:42:13.136123 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:13.136059    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8q5rf" event={"ID":"90bc69db-3427-4022-83ac-b134a6898421","Type":"ContainerStarted","Data":"f965eb1ad413fbb90d3f49b36d4bbe88dcf69d834acf89c207324bb35a1be406"}
Jul 31 01:42:13.136123 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:13.136094    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:42:13.162608 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:13.162531    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.440326076 podStartE2EDuration="11.1625199s" totalImagesPullingTime="7.699614604s" totalInitContainerRuntime="22.57922ms" isStatefulPod=false podCreationTimestamp="2026-07-31 01:42:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:42:13.161407073 +0000 UTC m=+291.527152699" watchObservedRunningTime="2026-07-31 01:42:13.1625199 +0000 UTC m=+291.528265525"
Jul 31 01:42:13.176352 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:13.176308    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8q5rf" podStartSLOduration=252.417340662 podStartE2EDuration="4m15.176292538s" totalImagesPullingTime="2.758951876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:42:13.175671417 +0000 UTC m=+291.541417065" watchObservedRunningTime="2026-07-31 01:42:13.176292538 +0000 UTC m=+291.542038166"
Jul 31 01:42:13.272843 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:13.272816    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:42:15.991876 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:15.991842    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:16.048410 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:16.048386    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:17.879709 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:17.879666    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:17.928934 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:17.928903    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:21.996696 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:21.996670    2556 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 01:42:22.900091 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:22.900054    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm" podUID="c2c45879-26e6-438b-bd0d-ff3ce513b92e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 31 01:42:23.142204 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:23.142173    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8q5rf"
Jul 31 01:42:27.118921 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:27.118864    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:27.182251 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:27.182220    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:30.693157 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:30.693132    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dxbfb_2b86f0a5-bacf-430e-b61b-21769d948195/init-textfile/0.log"
Jul 31 01:42:30.894126 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:30.894099    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dxbfb_2b86f0a5-bacf-430e-b61b-21769d948195/node-exporter/0.log"
Jul 31 01:42:31.093331 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:31.093304    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dxbfb_2b86f0a5-bacf-430e-b61b-21769d948195/kube-rbac-proxy/0.log"
Jul 31 01:42:31.892653 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:31.892622    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/init-config-reloader/0.log"
Jul 31 01:42:32.094151 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:32.094126    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/prometheus/0.log"
Jul 31 01:42:32.293112 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:32.293046    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/config-reloader/0.log"
Jul 31 01:42:32.493196 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:32.493171    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/thanos-sidecar/0.log"
Jul 31 01:42:32.693479 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:32.693453    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/kube-rbac-proxy-web/0.log"
Jul 31 01:42:32.877731 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:32.877702    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:32.893266 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:32.893244    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/kube-rbac-proxy/0.log"
Jul 31 01:42:32.899496 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:32.899470    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm" podUID="c2c45879-26e6-438b-bd0d-ff3ce513b92e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 31 01:42:32.899608 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:32.899534    2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm"
Jul 31 01:42:32.899967 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:32.899940    2556 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b6b687ade8f5e25fa0242dfdffd109a5110e57f78e6de85c3e304ccf5eaca5f3"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 31 01:42:32.900007 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:32.899987    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm" podUID="c2c45879-26e6-438b-bd0d-ff3ce513b92e" containerName="service-proxy" containerID="cri-o://b6b687ade8f5e25fa0242dfdffd109a5110e57f78e6de85c3e304ccf5eaca5f3" gracePeriod=30
Jul 31 01:42:32.979028 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:32.978962    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:33.093066 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:33.093046    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/kube-rbac-proxy-thanos/0.log"
Jul 31 01:42:33.207066 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:33.207037    2556 generic.go:356] "Generic (PLEG): container finished" podID="c2c45879-26e6-438b-bd0d-ff3ce513b92e" containerID="b6b687ade8f5e25fa0242dfdffd109a5110e57f78e6de85c3e304ccf5eaca5f3" exitCode=2
Jul 31 01:42:33.207254 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:33.207097    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm" event={"ID":"c2c45879-26e6-438b-bd0d-ff3ce513b92e","Type":"ContainerDied","Data":"b6b687ade8f5e25fa0242dfdffd109a5110e57f78e6de85c3e304ccf5eaca5f3"}
Jul 31 01:42:33.207254 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:33.207140    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d46556b44-qb5tm" event={"ID":"c2c45879-26e6-438b-bd0d-ff3ce513b92e","Type":"ContainerStarted","Data":"fabe6c6bccff9aab452f0fa6cf030609c474250c0bee0846aa24f1bc661082bc"}
Jul 31 01:42:33.693129 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:33.693099    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-lgpmv_b433e829-5435-43f5-912b-1b6a65e29be4/prometheus-operator-admission-webhook/0.log"
Jul 31 01:42:36.137118 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:36.137086    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:36.194517 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:36.194483    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:42:36.294525 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:36.294498    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-w6ktx_b1ea40d4-e2d3-46b7-b4a6-9d54bfefdb0f/download-server/0.log"
Jul 31 01:42:36.493923 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:36.493855    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8q5rf_90bc69db-3427-4022-83ac-b134a6898421/dns/0.log"
Jul 31 01:42:36.693078 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:36.693052    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8q5rf_90bc69db-3427-4022-83ac-b134a6898421/kube-rbac-proxy/0.log"
Jul 31 01:42:38.093711 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:38.093683    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tqppw_8525e366-5355-4406-8e20-88f336bf1e44/dns-node-resolver/0.log"
Jul 31 01:42:38.893198 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:42:38.893172    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s6rd8_144cf673-df86-42f6-9d5a-925a3fb98dec/serve-healthcheck-canary/0.log"
Jul 31 01:43:03.273320 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:03.273278    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:43:03.292497 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:03.292473    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:43:03.317053 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:03.317027    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 01:43:03.880019 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:03.879974    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:15.969015 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:15.968898    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:17.007402 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:17.007372    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:17.339533 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:17.339506    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:17.394676 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:17.394645    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:24.482732 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:24.482705    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:31.193359 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:31.193325    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:31.253752 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:31.253726    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:35.467447 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:35.467411    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:35.520213 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:35.520182    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:35.790406 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:35.790326    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:36.396795 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:36.396765    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:36.447853 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:36.447827    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:37.396456 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:37.396424    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:37.496785 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:37.496755    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:43.116938 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:43.116899    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:43.165474 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:43.165444    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:46.488870 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:46.488832    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:46.588647 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:46.588617    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:46.689711 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:46.689678    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:47.879518 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:47.879485    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:47.931840 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:47.931772    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:55.338296 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:55.338266    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:43:55.402012 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:43:55.401984    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:44:28.059922 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:44:28.059846    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:44:28.110139 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:44:28.110110    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:44:55.237491 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:44:55.237412    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:44:55.338421 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:44:55.338393    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:44:55.395457 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:44:55.395396    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:45:04.389078 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:45:04.389047    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:45:04.450651 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:45:04.450620    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:45:17.888824 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:45:17.888792    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:45:17.988491 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:45:17.988459    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:15.883484 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:15.883393    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:15.982212 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:15.982183    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:16.065379 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:16.065353    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:19.526263 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:19.526234    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:42.529268 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:42.529238    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:43.336498 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:43.336464    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:45.634866 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:45.634830    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:45.733317 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:45.733290    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:46.153740 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:46.153703    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:46.211659 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:46.211626    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:47.803392 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:47.803356    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:47.889804 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:47.889775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:48.371947 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:48.371914    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:48.474168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:48.474130    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:49.984403 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:49.984346    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:50.401253 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:50.401222    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:53.132269 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:53.132226    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:53.232974 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:53.232934    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:46:57.775361 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:46:57.775331    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:00.795888 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:00.795854    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:00.897276 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:00.897246    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:03.256600 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:03.256549    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:03.356093 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:03.356053    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:06.872794 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:06.872754    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:08.292860 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:08.292820    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:08.546474 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:08.546405    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:14.360621 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:14.360585    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:14.407380 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:14.407348    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:14.461134 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:14.461107    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:19.798397 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:19.798362    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:19.899506 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:19.899473    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:24.764016 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:24.763982    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:24.865683 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:24.865649    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"]
Jul 31 01:47:24.866889 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:24.866863    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:24.868701 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:24.868679    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:24.871255 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:24.871236    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 31 01:47:24.872337 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:24.872320    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 31 01:47:24.872426 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:24.872391    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 31 01:47:24.872426 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:24.872401    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5st4h\""
Jul 31 01:47:24.872531 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:24.872441    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 31 01:47:24.872531 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:24.872471    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 31 01:47:24.878403 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:24.878382    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"]
Jul 31 01:47:24.983576 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:24.983524    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:25.008160 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.008131    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/24cbb706-1704-4a27-a806-8e753e3bdc10-manager-config\") pod \"lws-controller-manager-8657cf6695-45dbm\" (UID: \"24cbb706-1704-4a27-a806-8e753e3bdc10\") " pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.008298 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.008178    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24cbb706-1704-4a27-a806-8e753e3bdc10-cert\") pod \"lws-controller-manager-8657cf6695-45dbm\" (UID: \"24cbb706-1704-4a27-a806-8e753e3bdc10\") " pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.008298 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.008217    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/24cbb706-1704-4a27-a806-8e753e3bdc10-metrics-cert\") pod \"lws-controller-manager-8657cf6695-45dbm\" (UID: \"24cbb706-1704-4a27-a806-8e753e3bdc10\") " pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.008298 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.008243    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkf75\" (UniqueName: \"kubernetes.io/projected/24cbb706-1704-4a27-a806-8e753e3bdc10-kube-api-access-xkf75\") pod \"lws-controller-manager-8657cf6695-45dbm\" (UID: \"24cbb706-1704-4a27-a806-8e753e3bdc10\") " pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.083820 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.083791    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:25.109409 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.109374    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/24cbb706-1704-4a27-a806-8e753e3bdc10-manager-config\") pod \"lws-controller-manager-8657cf6695-45dbm\" (UID: \"24cbb706-1704-4a27-a806-8e753e3bdc10\") " pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.109553 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.109419    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24cbb706-1704-4a27-a806-8e753e3bdc10-cert\") pod \"lws-controller-manager-8657cf6695-45dbm\" (UID: \"24cbb706-1704-4a27-a806-8e753e3bdc10\") " pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.109553 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.109444    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/24cbb706-1704-4a27-a806-8e753e3bdc10-metrics-cert\") pod \"lws-controller-manager-8657cf6695-45dbm\" (UID: \"24cbb706-1704-4a27-a806-8e753e3bdc10\") " pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.109553 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.109468    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkf75\" (UniqueName: \"kubernetes.io/projected/24cbb706-1704-4a27-a806-8e753e3bdc10-kube-api-access-xkf75\") pod \"lws-controller-manager-8657cf6695-45dbm\" (UID: \"24cbb706-1704-4a27-a806-8e753e3bdc10\") " pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.110156 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.110091    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/24cbb706-1704-4a27-a806-8e753e3bdc10-manager-config\") pod \"lws-controller-manager-8657cf6695-45dbm\" (UID: \"24cbb706-1704-4a27-a806-8e753e3bdc10\") " pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.112047 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.112025    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24cbb706-1704-4a27-a806-8e753e3bdc10-cert\") pod \"lws-controller-manager-8657cf6695-45dbm\" (UID: \"24cbb706-1704-4a27-a806-8e753e3bdc10\") " pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.112047 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.112038    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/24cbb706-1704-4a27-a806-8e753e3bdc10-metrics-cert\") pod \"lws-controller-manager-8657cf6695-45dbm\" (UID: \"24cbb706-1704-4a27-a806-8e753e3bdc10\") " pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.121102 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.121075    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkf75\" (UniqueName: \"kubernetes.io/projected/24cbb706-1704-4a27-a806-8e753e3bdc10-kube-api-access-xkf75\") pod \"lws-controller-manager-8657cf6695-45dbm\" (UID: \"24cbb706-1704-4a27-a806-8e753e3bdc10\") " pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.177275 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.177233    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:25.312956 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.312922    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"]
Jul 31 01:47:25.315752 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:47:25.315727    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24cbb706_1704_4a27_a806_8e753e3bdc10.slice/crio-6bfbe8cf78400f3587e031b79eacc79c8ed1c16bbd1da39388cfff252337da53 WatchSource:0}: Error finding container 6bfbe8cf78400f3587e031b79eacc79c8ed1c16bbd1da39388cfff252337da53: Status 404 returned error can't find the container with id 6bfbe8cf78400f3587e031b79eacc79c8ed1c16bbd1da39388cfff252337da53
Jul 31 01:47:25.317426 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.317414    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 01:47:25.591004 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.590954    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:25.839194 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.839159    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:25.947657 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.947578    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:25.971116 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.971089    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"]
Jul 31 01:47:25.975450 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.975427    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:25.977880 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.977846    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 31 01:47:25.978173 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.978158    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-r7hfn\""
Jul 31 01:47:25.978173 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.978178    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 31 01:47:25.978303 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.978176    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 31 01:47:25.978533 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.978519    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 31 01:47:25.989455 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:25.989428    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"]
Jul 31 01:47:26.084340 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.084304    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:26.119494 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.119466    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14d969da-0f92-4e5e-ab34-f2acc6e470ff-tmp\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.119667 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.119513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14d969da-0f92-4e5e-ab34-f2acc6e470ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.119667 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.119534    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14d969da-0f92-4e5e-ab34-f2acc6e470ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.119667 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.119630    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14d969da-0f92-4e5e-ab34-f2acc6e470ff-manifests\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.119771 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.119685    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxh85\" (UniqueName: \"kubernetes.io/projected/14d969da-0f92-4e5e-ab34-f2acc6e470ff-kube-api-access-jxh85\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.184817 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.184787    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:26.211668 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.211589    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm" event={"ID":"24cbb706-1704-4a27-a806-8e753e3bdc10","Type":"ContainerStarted","Data":"6bfbe8cf78400f3587e031b79eacc79c8ed1c16bbd1da39388cfff252337da53"}
Jul 31 01:47:26.221012 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.220982    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14d969da-0f92-4e5e-ab34-f2acc6e470ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.221190 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.221023    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14d969da-0f92-4e5e-ab34-f2acc6e470ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.221190 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.221048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14d969da-0f92-4e5e-ab34-f2acc6e470ff-manifests\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.221190 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.221088    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxh85\" (UniqueName: \"kubernetes.io/projected/14d969da-0f92-4e5e-ab34-f2acc6e470ff-kube-api-access-jxh85\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.221190 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.221128    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14d969da-0f92-4e5e-ab34-f2acc6e470ff-tmp\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.221696 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.221647    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14d969da-0f92-4e5e-ab34-f2acc6e470ff-manifests\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.223998 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.223975    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14d969da-0f92-4e5e-ab34-f2acc6e470ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.224178 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.224154    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14d969da-0f92-4e5e-ab34-f2acc6e470ff-tmp\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.224330 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.224315    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14d969da-0f92-4e5e-ab34-f2acc6e470ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.237856 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.237826    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxh85\" (UniqueName: \"kubernetes.io/projected/14d969da-0f92-4e5e-ab34-f2acc6e470ff-kube-api-access-jxh85\") pod \"opendatahub-operator-controller-manager-77df9dc58b-l7b9g\" (UID: \"14d969da-0f92-4e5e-ab34-f2acc6e470ff\") " pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.284904 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.284865    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:26.285806 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.285779    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:26.387124 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.387095    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:26.435293 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.435262    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"]
Jul 31 01:47:26.438847 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:47:26.438820    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d969da_0f92_4e5e_ab34_f2acc6e470ff.slice/crio-fcbbc9d94158b888907a5247657537e0646b4d15d63cb20e19fbb2a0918114a3 WatchSource:0}: Error finding container fcbbc9d94158b888907a5247657537e0646b4d15d63cb20e19fbb2a0918114a3: Status 404 returned error can't find the container with id fcbbc9d94158b888907a5247657537e0646b4d15d63cb20e19fbb2a0918114a3
Jul 31 01:47:26.491514 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:26.491431    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:27.219131 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:27.218993    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g" event={"ID":"14d969da-0f92-4e5e-ab34-f2acc6e470ff","Type":"ContainerStarted","Data":"fcbbc9d94158b888907a5247657537e0646b4d15d63cb20e19fbb2a0918114a3"}
Jul 31 01:47:31.237713 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:31.237678    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm" event={"ID":"24cbb706-1704-4a27-a806-8e753e3bdc10","Type":"ContainerStarted","Data":"4ed13f0525efcd909b856a3ce1bed5a34c39e6e22991443129183c712d0e4d9a"}
Jul 31 01:47:31.238138 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:31.237875    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:31.239262 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:31.239237    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g" event={"ID":"14d969da-0f92-4e5e-ab34-f2acc6e470ff","Type":"ContainerStarted","Data":"ce1ba88147b11dcc563658b869fcfab995cd381f5bdf916057bb96cc4d663d0b"}
Jul 31 01:47:31.254815 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:31.254771    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm" podStartSLOduration=2.28540014 podStartE2EDuration="7.254756362s" totalImagesPullingTime="4.969356222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:47:31.253801183 +0000 UTC m=+609.619546813" watchObservedRunningTime="2026-07-31 01:47:31.254756362 +0000 UTC m=+609.620501989"
Jul 31 01:47:32.244987 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:32.244950    2556 generic.go:356] "Generic (PLEG): container finished" podID="14d969da-0f92-4e5e-ab34-f2acc6e470ff" containerID="ce1ba88147b11dcc563658b869fcfab995cd381f5bdf916057bb96cc4d663d0b" exitCode=0
Jul 31 01:47:32.245374 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:32.245035    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g" event={"ID":"14d969da-0f92-4e5e-ab34-f2acc6e470ff","Type":"ContainerDied","Data":"ce1ba88147b11dcc563658b869fcfab995cd381f5bdf916057bb96cc4d663d0b"}
Jul 31 01:47:33.250545 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:33.250510    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g" event={"ID":"14d969da-0f92-4e5e-ab34-f2acc6e470ff","Type":"ContainerStarted","Data":"8e75532a4344b1f40bd45f54757c99af803bd70d34a97b1cf98573250752d193"}
Jul 31 01:47:33.250939 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:33.250660    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:33.270500 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:33.270444    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g" podStartSLOduration=2.986498615 podStartE2EDuration="8.270426252s" totalImagesPullingTime="4.196878873s" totalInitContainerRuntime="1.087048764s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:47:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:47:33.26834399 +0000 UTC m=+611.634089619" watchObservedRunningTime="2026-07-31 01:47:33.270426252 +0000 UTC m=+611.636171879"
Jul 31 01:47:37.969767 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:37.969731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:38.067734 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:38.067706    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:42.247179 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:42.247146    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8657cf6695-45dbm"
Jul 31 01:47:42.573284 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:42.573254    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:44.256199 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:44.256163    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77df9dc58b-l7b9g"
Jul 31 01:47:45.892708 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:45.892669    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:45.999632 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:45.999594    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:51.445671 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:51.445637    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:51.765500 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:51.765424    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:51.856214 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:51.856180    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:51.955882 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:51.955847    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:52.087994 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:52.087961    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:52.458267 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:52.458179    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:52.558408 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:52.558379    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:52.665921 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:52.665885    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:52.723740 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:52.723659    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:53.562146 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:53.562112    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:58.698657 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:58.698621    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:58.788204 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:58.788173    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:58.982196 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:58.982129    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"]
Jul 31 01:47:58.992593 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:58.992574    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:58.995095 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:58.995070    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 31 01:47:58.995095 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:58.995090    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 31 01:47:58.995449 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:58.995427    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 31 01:47:58.995573 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:58.995537    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ms5gd\""
Jul 31 01:47:58.995721 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:58.995671    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 31 01:47:58.995816 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:58.995793    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 01:47:58.996078 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:58.996061    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 31 01:47:58.997339 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:58.997320    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"]
Jul 31 01:47:59.075253 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.075217    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af296a8a-d9e7-4047-bef6-ad81aab47ddd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.075448 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.075268    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af296a8a-d9e7-4047-bef6-ad81aab47ddd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.075448 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.075294    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af296a8a-d9e7-4047-bef6-ad81aab47ddd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.075448 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.075315    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af296a8a-d9e7-4047-bef6-ad81aab47ddd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.075448 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.075344    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af296a8a-d9e7-4047-bef6-ad81aab47ddd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.075448 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.075367    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af296a8a-d9e7-4047-bef6-ad81aab47ddd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.075448 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.075391    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56qkx\" (UniqueName: \"kubernetes.io/projected/af296a8a-d9e7-4047-bef6-ad81aab47ddd-kube-api-access-56qkx\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.175701 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.175665    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af296a8a-d9e7-4047-bef6-ad81aab47ddd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.175701 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.175707    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af296a8a-d9e7-4047-bef6-ad81aab47ddd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.175928 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.175724    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af296a8a-d9e7-4047-bef6-ad81aab47ddd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.175928 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.175742    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af296a8a-d9e7-4047-bef6-ad81aab47ddd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.175928 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.175767    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af296a8a-d9e7-4047-bef6-ad81aab47ddd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.175928 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.175794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af296a8a-d9e7-4047-bef6-ad81aab47ddd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.175928 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.175906    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56qkx\" (UniqueName: \"kubernetes.io/projected/af296a8a-d9e7-4047-bef6-ad81aab47ddd-kube-api-access-56qkx\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.176663 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.176626    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af296a8a-d9e7-4047-bef6-ad81aab47ddd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.178261 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.178239    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af296a8a-d9e7-4047-bef6-ad81aab47ddd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.178363 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.178309    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af296a8a-d9e7-4047-bef6-ad81aab47ddd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.178697 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.178679    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af296a8a-d9e7-4047-bef6-ad81aab47ddd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.178835 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.178816    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af296a8a-d9e7-4047-bef6-ad81aab47ddd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.183666 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.183643    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af296a8a-d9e7-4047-bef6-ad81aab47ddd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.183875 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.183858    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56qkx\" (UniqueName: \"kubernetes.io/projected/af296a8a-d9e7-4047-bef6-ad81aab47ddd-kube-api-access-56qkx\") pod \"istiod-openshift-gateway-5677ccb49b-fjkxr\" (UID: \"af296a8a-d9e7-4047-bef6-ad81aab47ddd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.265216 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.265143    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:59.302801 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.302774    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:47:59.446136 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.446108    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"]
Jul 31 01:47:59.448411 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:47:59.448381    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf296a8a_d9e7_4047_bef6_ad81aab47ddd.slice/crio-a2c921a9ad7263f474f62aabeab1807a89c177f40ba6c97c92675063bec9d7d4 WatchSource:0}: Error finding container a2c921a9ad7263f474f62aabeab1807a89c177f40ba6c97c92675063bec9d7d4: Status 404 returned error can't find the container with id a2c921a9ad7263f474f62aabeab1807a89c177f40ba6c97c92675063bec9d7d4
Jul 31 01:47:59.529449 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.529376    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:47:59.652676 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:47:59.652646    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:48:00.351337 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:00.351298    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr" event={"ID":"af296a8a-d9e7-4047-bef6-ad81aab47ddd","Type":"ContainerStarted","Data":"a2c921a9ad7263f474f62aabeab1807a89c177f40ba6c97c92675063bec9d7d4"}
Jul 31 01:48:01.842744 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:01.842712    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:48:01.939290 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:01.939256    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:48:02.038934 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:02.038904    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:48:02.149214 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:02.149141    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:48:02.281678 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:02.281644    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:48:03.180583 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:03.180534    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 31 01:48:03.180866 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:03.180613    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 31 01:48:03.364089 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:03.364056    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr" event={"ID":"af296a8a-d9e7-4047-bef6-ad81aab47ddd","Type":"ContainerStarted","Data":"c3aa94c94acc4ceb77f760d5d2fb90ea25be07263bdcfc6f4c3d129eb7db2c0e"}
Jul 31 01:48:03.364240 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:03.364125    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:48:03.395347 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:03.395302    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr" podStartSLOduration=1.665419572 podStartE2EDuration="5.395286019s" totalImagesPullingTime="3.729866447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:48:03.394239768 +0000 UTC m=+641.759985396" watchObservedRunningTime="2026-07-31 01:48:03.395286019 +0000 UTC m=+641.761031647"
Jul 31 01:48:04.369738 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:04.369709    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fjkxr"
Jul 31 01:48:21.643065 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:21.643030    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:48:29.004482 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:29.004448    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:48:47.684082 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:47.684049    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:48:47.748123 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:47.748098    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:48:58.847159 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:58.847113    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:48:59.640419 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:48:59.640384    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:00.426751 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:00.426715    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:02.677407 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:02.677369    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:02.776386 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:02.776349    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:03.671955 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:03.671926    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:04.083357 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:04.083328    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:04.369354 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:04.369274    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:06.639781 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:06.639736    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:06.749335 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:06.749307    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:08.848532 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:08.848502    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:09.083149 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:09.083113    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:09.944160 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:09.944127    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:11.794599 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:11.794545    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:11.892156 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:11.892122    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:12.296338 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:12.296303    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:12.776267 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:12.776186    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:13.634065 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:13.634032    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:13.734639 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:13.734610    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:14.034322 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:14.034244    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:14.135022 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:14.134984    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:14.432168 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:14.432138    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:14.487836 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:14.487807    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:19.282528 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:19.282496    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:22.457463 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:22.457427    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:22.574630 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:22.574594    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:25.464679 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:25.464643    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:25.564065 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:25.564034    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:27.884056 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:27.884019    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:27.984183 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:27.984153    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:28.084330 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:28.084297    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:34.848603 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:34.848566    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:34.924721 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:34.924687    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:35.324326 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:35.324290    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:35.435803 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:35.435773    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:35.528843 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:35.528818    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:42.342482 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:42.342448    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:44.265697 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:44.265657    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:44.365536 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:44.365508    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:47.261417 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:47.261383    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:47.362516 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:47.362476    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:49.729196 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:49.729168    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:49.790695 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:49.790662    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:51.417185 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:51.417112    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:51.564980 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:51.564946    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:53.083324 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.083295    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"]
Jul 31 01:49:53.086760 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.086735    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"
Jul 31 01:49:53.086892 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.086853    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:53.089030 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.088997    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 31 01:49:53.089158 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.089032    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kp5l8\""
Jul 31 01:49:53.089158 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.089099    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 31 01:49:53.089616 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.089601    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 31 01:49:53.101277 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.101252    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"]
Jul 31 01:49:53.180382 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.180356    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"]
Jul 31 01:49:53.183481 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.183457    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:53.246071 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.246045    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:53.264266 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.264243    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rsp9\" (UniqueName: \"kubernetes.io/projected/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128-kube-api-access-2rsp9\") pod \"limitador-limitador-859c4d96d6-zhnv5\" (UID: \"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"
Jul 31 01:49:53.264343 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.264271    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128-config-file\") pod \"limitador-limitador-859c4d96d6-zhnv5\" (UID: \"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"
Jul 31 01:49:53.365261 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.365194    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rsp9\" (UniqueName: \"kubernetes.io/projected/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128-kube-api-access-2rsp9\") pod \"limitador-limitador-859c4d96d6-zhnv5\" (UID: \"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"
Jul 31 01:49:53.365261 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.365230    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128-config-file\") pod \"limitador-limitador-859c4d96d6-zhnv5\" (UID: \"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"
Jul 31 01:49:53.365829 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.365806    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128-config-file\") pod \"limitador-limitador-859c4d96d6-zhnv5\" (UID: \"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"
Jul 31 01:49:53.372393 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.372372    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rsp9\" (UniqueName: \"kubernetes.io/projected/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128-kube-api-access-2rsp9\") pod \"limitador-limitador-859c4d96d6-zhnv5\" (UID: \"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"
Jul 31 01:49:53.398378 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.398357    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"
Jul 31 01:49:53.525646 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.525618    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"]
Jul 31 01:49:53.529248 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:49:53.529218    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef33857_cbc3_4b73_8f9f_06fd5fc5e128.slice/crio-cfc1f6cdb8b2f3dd66ba4ccd9646e68f55c281f6b7a2592869d7cf63585ad201 WatchSource:0}: Error finding container cfc1f6cdb8b2f3dd66ba4ccd9646e68f55c281f6b7a2592869d7cf63585ad201: Status 404 returned error can't find the container with id cfc1f6cdb8b2f3dd66ba4ccd9646e68f55c281f6b7a2592869d7cf63585ad201
Jul 31 01:49:53.752593 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.752506    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5" event={"ID":"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128","Type":"ContainerStarted","Data":"cfc1f6cdb8b2f3dd66ba4ccd9646e68f55c281f6b7a2592869d7cf63585ad201"}
Jul 31 01:49:53.911380 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:53.911347    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:54.014290 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:54.014220    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:49:57.770402 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:57.770365    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5" event={"ID":"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128","Type":"ContainerStarted","Data":"7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49"}
Jul 31 01:49:57.770827 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:57.770599    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"
Jul 31 01:49:57.785045 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:49:57.784999    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5" podStartSLOduration=1.605793419 podStartE2EDuration="4.78498653s" totalImagesPullingTime="3.179193111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:49:57.783856417 +0000 UTC m=+756.149602043" watchObservedRunningTime="2026-07-31 01:49:57.78498653 +0000 UTC m=+756.150732172"
Jul 31 01:50:08.040081 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.040049    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"]
Jul 31 01:50:08.040647 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.040209    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5" podUID="3ef33857-cbc3-4b73-8f9f-06fd5fc5e128" containerName="limitador" containerID="cri-o://7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49" gracePeriod=30
Jul 31 01:50:08.040958 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.040926    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"
Jul 31 01:50:08.581754 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.581731    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"
Jul 31 01:50:08.681772 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.681700    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128-config-file\") pod \"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128\" (UID: \"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128\") "
Jul 31 01:50:08.681772 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.681751    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rsp9\" (UniqueName: \"kubernetes.io/projected/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128-kube-api-access-2rsp9\") pod \"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128\" (UID: \"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128\") "
Jul 31 01:50:08.682069 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.682048    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128-config-file" (OuterVolumeSpecName: "config-file") pod "3ef33857-cbc3-4b73-8f9f-06fd5fc5e128" (UID: "3ef33857-cbc3-4b73-8f9f-06fd5fc5e128"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 01:50:08.684043 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.684022    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128-kube-api-access-2rsp9" (OuterVolumeSpecName: "kube-api-access-2rsp9") pod "3ef33857-cbc3-4b73-8f9f-06fd5fc5e128" (UID: "3ef33857-cbc3-4b73-8f9f-06fd5fc5e128"). InnerVolumeSpecName "kube-api-access-2rsp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 01:50:08.782537 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.782500    2556 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128-config-file\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 01:50:08.782537 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.782530    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rsp9\" (UniqueName: \"kubernetes.io/projected/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128-kube-api-access-2rsp9\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 01:50:08.810513 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.810481    2556 generic.go:356] "Generic (PLEG): container finished" podID="3ef33857-cbc3-4b73-8f9f-06fd5fc5e128" containerID="7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49" exitCode=0
Jul 31 01:50:08.810680 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.810537    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"
Jul 31 01:50:08.810680 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.810588    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5" event={"ID":"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128","Type":"ContainerDied","Data":"7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49"}
Jul 31 01:50:08.810680 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.810665    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zhnv5" event={"ID":"3ef33857-cbc3-4b73-8f9f-06fd5fc5e128","Type":"ContainerDied","Data":"cfc1f6cdb8b2f3dd66ba4ccd9646e68f55c281f6b7a2592869d7cf63585ad201"}
Jul 31 01:50:08.810827 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.810688    2556 scope.go:117] "RemoveContainer" containerID="7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49"
Jul 31 01:50:08.819077 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.819060    2556 scope.go:117] "RemoveContainer" containerID="7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49"
Jul 31 01:50:08.819326 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:50:08.819308    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49\": container with ID starting with 7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49 not found: ID does not exist" containerID="7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49"
Jul 31 01:50:08.819372 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.819334    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49"} err="failed to get container status \"7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49\": rpc error: code = NotFound desc = could not find container \"7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49\": container with ID starting with 7bd4092af63a6f7f90dd37e200664746fbb4296eec3993dd68f6975d14105f49 not found: ID does not exist"
Jul 31 01:50:08.829687 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.829664    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"]
Jul 31 01:50:08.834077 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:08.834057    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zhnv5"]
Jul 31 01:50:09.267221 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.267189    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:50:09.368656 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.368621    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:50:09.714550 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.714525    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"]
Jul 31 01:50:09.714818 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.714805    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ef33857-cbc3-4b73-8f9f-06fd5fc5e128" containerName="limitador"
Jul 31 01:50:09.714860 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.714820    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ef33857-cbc3-4b73-8f9f-06fd5fc5e128" containerName="limitador"
Jul 31 01:50:09.714892 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.714872    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ef33857-cbc3-4b73-8f9f-06fd5fc5e128" containerName="limitador"
Jul 31 01:50:09.718995 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.718967    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"
Jul 31 01:50:09.721300 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.721276    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fjw9c\""
Jul 31 01:50:09.721422 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.721283    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 01:50:09.721422 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.721318    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 31 01:50:09.722044 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.722029    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 31 01:50:09.726365 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.726344    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"]
Jul 31 01:50:09.751251 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.751227    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:50:09.788093 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.788067    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhh47\" (UniqueName: \"kubernetes.io/projected/8ffafa2b-6ae5-4a3d-90f3-469e80316950-kube-api-access-rhh47\") pod \"postgres-69fb4594cf-dqzch\" (UID: \"8ffafa2b-6ae5-4a3d-90f3-469e80316950\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"
Jul 31 01:50:09.788200 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.788127    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8ffafa2b-6ae5-4a3d-90f3-469e80316950-data\") pod \"postgres-69fb4594cf-dqzch\" (UID: \"8ffafa2b-6ae5-4a3d-90f3-469e80316950\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"
Jul 31 01:50:09.889135 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.889107    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8ffafa2b-6ae5-4a3d-90f3-469e80316950-data\") pod \"postgres-69fb4594cf-dqzch\" (UID: \"8ffafa2b-6ae5-4a3d-90f3-469e80316950\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"
Jul 31 01:50:09.889279 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.889149    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhh47\" (UniqueName: \"kubernetes.io/projected/8ffafa2b-6ae5-4a3d-90f3-469e80316950-kube-api-access-rhh47\") pod \"postgres-69fb4594cf-dqzch\" (UID: \"8ffafa2b-6ae5-4a3d-90f3-469e80316950\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"
Jul 31 01:50:09.889449 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.889433    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8ffafa2b-6ae5-4a3d-90f3-469e80316950-data\") pod \"postgres-69fb4594cf-dqzch\" (UID: \"8ffafa2b-6ae5-4a3d-90f3-469e80316950\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"
Jul 31 01:50:09.896747 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:09.896727    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhh47\" (UniqueName: \"kubernetes.io/projected/8ffafa2b-6ae5-4a3d-90f3-469e80316950-kube-api-access-rhh47\") pod \"postgres-69fb4594cf-dqzch\" (UID: \"8ffafa2b-6ae5-4a3d-90f3-469e80316950\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"
Jul 31 01:50:10.028855 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:10.028781    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"
Jul 31 01:50:10.150210 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:10.150179    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ef33857-cbc3-4b73-8f9f-06fd5fc5e128" path="/var/lib/kubelet/pods/3ef33857-cbc3-4b73-8f9f-06fd5fc5e128/volumes"
Jul 31 01:50:10.150499 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:10.150483    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"]
Jul 31 01:50:10.153601 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:50:10.153570    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffafa2b_6ae5_4a3d_90f3_469e80316950.slice/crio-8c83aa29db737ba22b324b672afd05422615ec87564b5350ca55723705818c19 WatchSource:0}: Error finding container 8c83aa29db737ba22b324b672afd05422615ec87564b5350ca55723705818c19: Status 404 returned error can't find the container with id 8c83aa29db737ba22b324b672afd05422615ec87564b5350ca55723705818c19
Jul 31 01:50:10.820069 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:10.820030    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch" event={"ID":"8ffafa2b-6ae5-4a3d-90f3-469e80316950","Type":"ContainerStarted","Data":"8c83aa29db737ba22b324b672afd05422615ec87564b5350ca55723705818c19"}
Jul 31 01:50:22.581039 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:22.581019    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 31 01:50:22.871295 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:22.871265    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch" event={"ID":"8ffafa2b-6ae5-4a3d-90f3-469e80316950","Type":"ContainerStarted","Data":"16ac908c7a1c3a4a3266b3d30da37f725c16052e3c75374f28fd1303283801bb"}
Jul 31 01:50:22.871476 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:22.871383    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"
Jul 31 01:50:22.886880 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:22.886837    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch" podStartSLOduration=1.463050798 podStartE2EDuration="13.886823508s" totalImagesPullingTime="12.42377271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:50:22.885122101 +0000 UTC m=+781.250867728" watchObservedRunningTime="2026-07-31 01:50:22.886823508 +0000 UTC m=+781.252569135"
Jul 31 01:50:28.904229 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:28.904197    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dqzch"
Jul 31 01:50:29.216200 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:29.216118    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:50:29.533171 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:29.533086    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:50:36.029184 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.029136    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:50:36.039429 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.039402    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c96c8fb6c-fnrj7"]
Jul 31 01:50:36.042818 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.042803    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:36.045049 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.045028    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 31 01:50:36.045323 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.045306    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gwhc5\""
Jul 31 01:50:36.045373 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.045314    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 31 01:50:36.056771 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.056751    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c96c8fb6c-fnrj7"]
Jul 31 01:50:36.100613 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.100577    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d42e885-71ad-4bcd-816e-ed69e0be13b5-webhook-cert\") pod \"maas-controller-c96c8fb6c-fnrj7\" (UID: \"2d42e885-71ad-4bcd-816e-ed69e0be13b5\") " pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:36.100764 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.100641    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-725ck\" (UniqueName: \"kubernetes.io/projected/2d42e885-71ad-4bcd-816e-ed69e0be13b5-kube-api-access-725ck\") pod \"maas-controller-c96c8fb6c-fnrj7\" (UID: \"2d42e885-71ad-4bcd-816e-ed69e0be13b5\") " pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:36.125744 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.125710    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:50:36.189622 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.189580    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:50:36.201406 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.201387    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-725ck\" (UniqueName: \"kubernetes.io/projected/2d42e885-71ad-4bcd-816e-ed69e0be13b5-kube-api-access-725ck\") pod \"maas-controller-c96c8fb6c-fnrj7\" (UID: \"2d42e885-71ad-4bcd-816e-ed69e0be13b5\") " pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:36.201548 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.201428    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d42e885-71ad-4bcd-816e-ed69e0be13b5-webhook-cert\") pod \"maas-controller-c96c8fb6c-fnrj7\" (UID: \"2d42e885-71ad-4bcd-816e-ed69e0be13b5\") " pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:36.201548 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:50:36.201524    2556 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 31 01:50:36.201689 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:50:36.201601    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d42e885-71ad-4bcd-816e-ed69e0be13b5-webhook-cert podName:2d42e885-71ad-4bcd-816e-ed69e0be13b5 nodeName:}" failed. No retries permitted until 2026-07-31 01:50:36.701585582 +0000 UTC m=+795.067331187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2d42e885-71ad-4bcd-816e-ed69e0be13b5-webhook-cert") pod "maas-controller-c96c8fb6c-fnrj7" (UID: "2d42e885-71ad-4bcd-816e-ed69e0be13b5") : secret "maas-controller-webhook-cert" not found
Jul 31 01:50:36.213702 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.213685    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-725ck\" (UniqueName: \"kubernetes.io/projected/2d42e885-71ad-4bcd-816e-ed69e0be13b5-kube-api-access-725ck\") pod \"maas-controller-c96c8fb6c-fnrj7\" (UID: \"2d42e885-71ad-4bcd-816e-ed69e0be13b5\") " pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:36.227023 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.226997    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7745c54f68-sqzvv"]
Jul 31 01:50:36.230057 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.230041    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7745c54f68-sqzvv"
Jul 31 01:50:36.241574 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.241536    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7745c54f68-sqzvv"]
Jul 31 01:50:36.272539 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.272474    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:50:36.302747 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.302695    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63f0d476-d27a-4007-80da-df54354ca177-webhook-cert\") pod \"maas-controller-7745c54f68-sqzvv\" (UID: \"63f0d476-d27a-4007-80da-df54354ca177\") " pod="opendatahub/maas-controller-7745c54f68-sqzvv"
Jul 31 01:50:36.302747 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.302737    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cxnh\" (UniqueName: \"kubernetes.io/projected/63f0d476-d27a-4007-80da-df54354ca177-kube-api-access-7cxnh\") pod \"maas-controller-7745c54f68-sqzvv\" (UID: \"63f0d476-d27a-4007-80da-df54354ca177\") " pod="opendatahub/maas-controller-7745c54f68-sqzvv"
Jul 31 01:50:36.403715 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.403671    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63f0d476-d27a-4007-80da-df54354ca177-webhook-cert\") pod \"maas-controller-7745c54f68-sqzvv\" (UID: \"63f0d476-d27a-4007-80da-df54354ca177\") " pod="opendatahub/maas-controller-7745c54f68-sqzvv"
Jul 31 01:50:36.403849 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.403736    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxnh\" (UniqueName: \"kubernetes.io/projected/63f0d476-d27a-4007-80da-df54354ca177-kube-api-access-7cxnh\") pod \"maas-controller-7745c54f68-sqzvv\" (UID: \"63f0d476-d27a-4007-80da-df54354ca177\") " pod="opendatahub/maas-controller-7745c54f68-sqzvv"
Jul 31 01:50:36.406155 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.406137    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63f0d476-d27a-4007-80da-df54354ca177-webhook-cert\") pod \"maas-controller-7745c54f68-sqzvv\" (UID: \"63f0d476-d27a-4007-80da-df54354ca177\") " pod="opendatahub/maas-controller-7745c54f68-sqzvv"
Jul 31 01:50:36.411213 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.411191    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxnh\" (UniqueName: \"kubernetes.io/projected/63f0d476-d27a-4007-80da-df54354ca177-kube-api-access-7cxnh\") pod \"maas-controller-7745c54f68-sqzvv\" (UID: \"63f0d476-d27a-4007-80da-df54354ca177\") " pod="opendatahub/maas-controller-7745c54f68-sqzvv"
Jul 31 01:50:36.539859 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.539825    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7745c54f68-sqzvv"
Jul 31 01:50:36.664622 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.664591    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7745c54f68-sqzvv"]
Jul 31 01:50:36.667935 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:50:36.667911    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f0d476_d27a_4007_80da_df54354ca177.slice/crio-b9e8025e5cc44697dbce2ffaa8ad504d6c6602e882ca95b516c5bee2f530a24a WatchSource:0}: Error finding container b9e8025e5cc44697dbce2ffaa8ad504d6c6602e882ca95b516c5bee2f530a24a: Status 404 returned error can't find the container with id b9e8025e5cc44697dbce2ffaa8ad504d6c6602e882ca95b516c5bee2f530a24a
Jul 31 01:50:36.706940 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.706916    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d42e885-71ad-4bcd-816e-ed69e0be13b5-webhook-cert\") pod \"maas-controller-c96c8fb6c-fnrj7\" (UID: \"2d42e885-71ad-4bcd-816e-ed69e0be13b5\") " pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:36.709289 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.709271    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d42e885-71ad-4bcd-816e-ed69e0be13b5-webhook-cert\") pod \"maas-controller-c96c8fb6c-fnrj7\" (UID: \"2d42e885-71ad-4bcd-816e-ed69e0be13b5\") " pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:36.925411 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.925337    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7745c54f68-sqzvv" event={"ID":"63f0d476-d27a-4007-80da-df54354ca177","Type":"ContainerStarted","Data":"b9e8025e5cc44697dbce2ffaa8ad504d6c6602e882ca95b516c5bee2f530a24a"}
Jul 31 01:50:36.952462 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:36.952438    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:37.074924 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:37.074902    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c96c8fb6c-fnrj7"]
Jul 31 01:50:37.077271 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:50:37.077248    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d42e885_71ad_4bcd_816e_ed69e0be13b5.slice/crio-973a99e21a35e9ff7bc799e9eaebd718f9c9dae05650e1469a47075ddef53aba WatchSource:0}: Error finding container 973a99e21a35e9ff7bc799e9eaebd718f9c9dae05650e1469a47075ddef53aba: Status 404 returned error can't find the container with id 973a99e21a35e9ff7bc799e9eaebd718f9c9dae05650e1469a47075ddef53aba
Jul 31 01:50:37.932547 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:37.932512    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c96c8fb6c-fnrj7" event={"ID":"2d42e885-71ad-4bcd-816e-ed69e0be13b5","Type":"ContainerStarted","Data":"973a99e21a35e9ff7bc799e9eaebd718f9c9dae05650e1469a47075ddef53aba"}
Jul 31 01:50:39.259872 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:39.259829    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:50:39.361595 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:39.361511    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:50:39.942198 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:39.942155    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7745c54f68-sqzvv" event={"ID":"63f0d476-d27a-4007-80da-df54354ca177","Type":"ContainerStarted","Data":"ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b"}
Jul 31 01:50:39.942407 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:39.942260    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7745c54f68-sqzvv"
Jul 31 01:50:39.943650 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:39.943629    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c96c8fb6c-fnrj7" event={"ID":"2d42e885-71ad-4bcd-816e-ed69e0be13b5","Type":"ContainerStarted","Data":"6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c"}
Jul 31 01:50:39.943770 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:39.943756    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:39.958060 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:39.958023    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7745c54f68-sqzvv" podStartSLOduration=1.543864229 podStartE2EDuration="3.958011369s" totalImagesPullingTime="2.41414714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:50:39.956618554 +0000 UTC m=+798.322364181" watchObservedRunningTime="2026-07-31 01:50:39.958011369 +0000 UTC m=+798.323756995"
Jul 31 01:50:39.971083 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:39.971035    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c96c8fb6c-fnrj7" podStartSLOduration=1.960376809 podStartE2EDuration="3.971023939s" totalImagesPullingTime="2.01064713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:50:39.969885166 +0000 UTC m=+798.335630794" watchObservedRunningTime="2026-07-31 01:50:39.971023939 +0000 UTC m=+798.336769565"
Jul 31 01:50:50.949576 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:50.949488    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7745c54f68-sqzvv"
Jul 31 01:50:50.949576 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:50.949573    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:51.007905 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.007874    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c96c8fb6c-fnrj7"]
Jul 31 01:50:51.008099 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.008074    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c96c8fb6c-fnrj7" podUID="2d42e885-71ad-4bcd-816e-ed69e0be13b5" containerName="manager" containerID="cri-o://6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c" gracePeriod=30
Jul 31 01:50:51.256831 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.256803    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:51.427773 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.427742    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-725ck\" (UniqueName: \"kubernetes.io/projected/2d42e885-71ad-4bcd-816e-ed69e0be13b5-kube-api-access-725ck\") pod \"2d42e885-71ad-4bcd-816e-ed69e0be13b5\" (UID: \"2d42e885-71ad-4bcd-816e-ed69e0be13b5\") "
Jul 31 01:50:51.427943 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.427803    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d42e885-71ad-4bcd-816e-ed69e0be13b5-webhook-cert\") pod \"2d42e885-71ad-4bcd-816e-ed69e0be13b5\" (UID: \"2d42e885-71ad-4bcd-816e-ed69e0be13b5\") "
Jul 31 01:50:51.429978 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.429948    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d42e885-71ad-4bcd-816e-ed69e0be13b5-kube-api-access-725ck" (OuterVolumeSpecName: "kube-api-access-725ck") pod "2d42e885-71ad-4bcd-816e-ed69e0be13b5" (UID: "2d42e885-71ad-4bcd-816e-ed69e0be13b5"). InnerVolumeSpecName "kube-api-access-725ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 01:50:51.429978 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.429948    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d42e885-71ad-4bcd-816e-ed69e0be13b5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2d42e885-71ad-4bcd-816e-ed69e0be13b5" (UID: "2d42e885-71ad-4bcd-816e-ed69e0be13b5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 01:50:51.528917 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.528849    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-725ck\" (UniqueName: \"kubernetes.io/projected/2d42e885-71ad-4bcd-816e-ed69e0be13b5-kube-api-access-725ck\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 01:50:51.528917 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.528875    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d42e885-71ad-4bcd-816e-ed69e0be13b5-webhook-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 01:50:51.989446 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.989411    2556 generic.go:356] "Generic (PLEG): container finished" podID="2d42e885-71ad-4bcd-816e-ed69e0be13b5" containerID="6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c" exitCode=0
Jul 31 01:50:51.989938 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.989462    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c96c8fb6c-fnrj7"
Jul 31 01:50:51.989938 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.989479    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c96c8fb6c-fnrj7" event={"ID":"2d42e885-71ad-4bcd-816e-ed69e0be13b5","Type":"ContainerDied","Data":"6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c"}
Jul 31 01:50:51.989938 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.989521    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c96c8fb6c-fnrj7" event={"ID":"2d42e885-71ad-4bcd-816e-ed69e0be13b5","Type":"ContainerDied","Data":"973a99e21a35e9ff7bc799e9eaebd718f9c9dae05650e1469a47075ddef53aba"}
Jul 31 01:50:51.989938 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.989543    2556 scope.go:117] "RemoveContainer" containerID="6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c"
Jul 31 01:50:51.997653 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.997633    2556 scope.go:117] "RemoveContainer" containerID="6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c"
Jul 31 01:50:51.997908 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:50:51.997888    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c\": container with ID starting with 6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c not found: ID does not exist" containerID="6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c"
Jul 31 01:50:51.997970 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:51.997916    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c"} err="failed to get container status \"6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c\": rpc error: code = NotFound desc = could not find container \"6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c\": container with ID starting with 6d1270b62b3ed5ced2e8713b2b4b56e91be10f0cd0df61c838e4c38ab7c8e53c not found: ID does not exist"
Jul 31 01:50:52.008071 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:52.008051    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c96c8fb6c-fnrj7"]
Jul 31 01:50:52.009903 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:52.009884    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c96c8fb6c-fnrj7"]
Jul 31 01:50:52.150368 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:50:52.150340    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d42e885-71ad-4bcd-816e-ed69e0be13b5" path="/var/lib/kubelet/pods/2d42e885-71ad-4bcd-816e-ed69e0be13b5/volumes"
Jul 31 01:51:10.870455 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:10.870418    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:51:10.983664 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:10.983624    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:51:12.456861 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.456825    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:51:12.513737 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.513705    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"]
Jul 31 01:51:12.513988 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.513977    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d42e885-71ad-4bcd-816e-ed69e0be13b5" containerName="manager"
Jul 31 01:51:12.514028 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.513989    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d42e885-71ad-4bcd-816e-ed69e0be13b5" containerName="manager"
Jul 31 01:51:12.514061 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.514042    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d42e885-71ad-4bcd-816e-ed69e0be13b5" containerName="manager"
Jul 31 01:51:12.534037 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.534009    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"]
Jul 31 01:51:12.534179 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.534098    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:12.536463 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.536439    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 31 01:51:12.536603 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.536484    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9mqr2\""
Jul 31 01:51:12.586626 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.586589    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnlcn\" (UniqueName: \"kubernetes.io/projected/254c4573-cac4-45e1-9368-11a24bb2841e-kube-api-access-wnlcn\") pod \"maas-api-77f76cbbbd-27pnj\" (UID: \"254c4573-cac4-45e1-9368-11a24bb2841e\") " pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:12.586626 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.586621    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c4573-cac4-45e1-9368-11a24bb2841e-openshift-service-ca\") pod \"maas-api-77f76cbbbd-27pnj\" (UID: \"254c4573-cac4-45e1-9368-11a24bb2841e\") " pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:12.586804 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.586640    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254c4573-cac4-45e1-9368-11a24bb2841e-maas-api-tls\") pod \"maas-api-77f76cbbbd-27pnj\" (UID: \"254c4573-cac4-45e1-9368-11a24bb2841e\") " pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:12.602370 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.602344    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:51:12.687373 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.687335    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnlcn\" (UniqueName: \"kubernetes.io/projected/254c4573-cac4-45e1-9368-11a24bb2841e-kube-api-access-wnlcn\") pod \"maas-api-77f76cbbbd-27pnj\" (UID: \"254c4573-cac4-45e1-9368-11a24bb2841e\") " pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:12.687373 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.687379    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c4573-cac4-45e1-9368-11a24bb2841e-openshift-service-ca\") pod \"maas-api-77f76cbbbd-27pnj\" (UID: \"254c4573-cac4-45e1-9368-11a24bb2841e\") " pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:12.687651 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.687496    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254c4573-cac4-45e1-9368-11a24bb2841e-maas-api-tls\") pod \"maas-api-77f76cbbbd-27pnj\" (UID: \"254c4573-cac4-45e1-9368-11a24bb2841e\") " pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:12.688019 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.687997    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c4573-cac4-45e1-9368-11a24bb2841e-openshift-service-ca\") pod \"maas-api-77f76cbbbd-27pnj\" (UID: \"254c4573-cac4-45e1-9368-11a24bb2841e\") " pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:12.689937 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.689911    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254c4573-cac4-45e1-9368-11a24bb2841e-maas-api-tls\") pod \"maas-api-77f76cbbbd-27pnj\" (UID: \"254c4573-cac4-45e1-9368-11a24bb2841e\") " pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:12.694918 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.694896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnlcn\" (UniqueName: \"kubernetes.io/projected/254c4573-cac4-45e1-9368-11a24bb2841e-kube-api-access-wnlcn\") pod \"maas-api-77f76cbbbd-27pnj\" (UID: \"254c4573-cac4-45e1-9368-11a24bb2841e\") " pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:12.844628 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.844600    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:12.978200 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:12.978128    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"]
Jul 31 01:51:12.982634 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:51:12.982606    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254c4573_cac4_45e1_9368_11a24bb2841e.slice/crio-a08f73d3f26e1108894c2da175f4ee58dafa5f2d43801c75ef630c77ea037b1d WatchSource:0}: Error finding container a08f73d3f26e1108894c2da175f4ee58dafa5f2d43801c75ef630c77ea037b1d: Status 404 returned error can't find the container with id a08f73d3f26e1108894c2da175f4ee58dafa5f2d43801c75ef630c77ea037b1d
Jul 31 01:51:13.004598 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.004537    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:51:13.079923 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.079890    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj" event={"ID":"254c4573-cac4-45e1-9368-11a24bb2841e","Type":"ContainerStarted","Data":"a08f73d3f26e1108894c2da175f4ee58dafa5f2d43801c75ef630c77ea037b1d"}
Jul 31 01:51:13.109093 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.109028    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"]
Jul 31 01:51:13.111646 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.111622    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:51:13.113451 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.113437    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"
Jul 31 01:51:13.115533 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.115514    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 31 01:51:13.115654 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.115614    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-c4v62\""
Jul 31 01:51:13.120918 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.120895    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"]
Jul 31 01:51:13.141779 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.141758    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-l2mkg"]
Jul 31 01:51:13.144800 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.144786    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg"
Jul 31 01:51:13.156837 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.156816    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-l2mkg"]
Jul 31 01:51:13.191546 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.191520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjx8\" (UniqueName: \"kubernetes.io/projected/fa01e1a2-7436-45ba-b058-346e7534e8ec-kube-api-access-bqjx8\") pod \"payload-processing-5cc6cd444b-l2mkg\" (UID: \"fa01e1a2-7436-45ba-b058-346e7534e8ec\") " pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg"
Jul 31 01:51:13.191692 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.191577    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66c5894-f7c2-4e5a-8a9c-ecd308c986df-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-wfhq9\" (UID: \"e66c5894-f7c2-4e5a-8a9c-ecd308c986df\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"
Jul 31 01:51:13.191692 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.191670    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa01e1a2-7436-45ba-b058-346e7534e8ec-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-l2mkg\" (UID: \"fa01e1a2-7436-45ba-b058-346e7534e8ec\") " pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg"
Jul 31 01:51:13.191768 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.191695    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nk8w\" (UniqueName: \"kubernetes.io/projected/e66c5894-f7c2-4e5a-8a9c-ecd308c986df-kube-api-access-5nk8w\") pod \"payload-pre-processing-687dbb6cf6-wfhq9\" (UID: \"e66c5894-f7c2-4e5a-8a9c-ecd308c986df\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"
Jul 31 01:51:13.293082 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.293042    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjx8\" (UniqueName: \"kubernetes.io/projected/fa01e1a2-7436-45ba-b058-346e7534e8ec-kube-api-access-bqjx8\") pod \"payload-processing-5cc6cd444b-l2mkg\" (UID: \"fa01e1a2-7436-45ba-b058-346e7534e8ec\") " pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg"
Jul 31 01:51:13.293252 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.293097    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66c5894-f7c2-4e5a-8a9c-ecd308c986df-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-wfhq9\" (UID: \"e66c5894-f7c2-4e5a-8a9c-ecd308c986df\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"
Jul 31 01:51:13.293252 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.293149    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa01e1a2-7436-45ba-b058-346e7534e8ec-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-l2mkg\" (UID: \"fa01e1a2-7436-45ba-b058-346e7534e8ec\") " pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg"
Jul 31 01:51:13.293252 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.293173    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nk8w\" (UniqueName: \"kubernetes.io/projected/e66c5894-f7c2-4e5a-8a9c-ecd308c986df-kube-api-access-5nk8w\") pod \"payload-pre-processing-687dbb6cf6-wfhq9\" (UID: \"e66c5894-f7c2-4e5a-8a9c-ecd308c986df\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"
Jul 31 01:51:13.293748 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.293729    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66c5894-f7c2-4e5a-8a9c-ecd308c986df-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-wfhq9\" (UID: \"e66c5894-f7c2-4e5a-8a9c-ecd308c986df\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"
Jul 31 01:51:13.293818 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.293790    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa01e1a2-7436-45ba-b058-346e7534e8ec-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-l2mkg\" (UID: \"fa01e1a2-7436-45ba-b058-346e7534e8ec\") " pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg"
Jul 31 01:51:13.300141 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.300108    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nk8w\" (UniqueName: \"kubernetes.io/projected/e66c5894-f7c2-4e5a-8a9c-ecd308c986df-kube-api-access-5nk8w\") pod \"payload-pre-processing-687dbb6cf6-wfhq9\" (UID: \"e66c5894-f7c2-4e5a-8a9c-ecd308c986df\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"
Jul 31 01:51:13.300254 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.300237    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjx8\" (UniqueName: \"kubernetes.io/projected/fa01e1a2-7436-45ba-b058-346e7534e8ec-kube-api-access-bqjx8\") pod \"payload-processing-5cc6cd444b-l2mkg\" (UID: \"fa01e1a2-7436-45ba-b058-346e7534e8ec\") " pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg"
Jul 31 01:51:13.425835 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.425763    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"
Jul 31 01:51:13.453995 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.453969    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg"
Jul 31 01:51:13.566839 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.566815    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"]
Jul 31 01:51:13.569881 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:51:13.569848    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66c5894_f7c2_4e5a_8a9c_ecd308c986df.slice/crio-989f2222b482d1f28900b119a2a7f31684143f3dd59d1c6701cfcb7f3c62b106 WatchSource:0}: Error finding container 989f2222b482d1f28900b119a2a7f31684143f3dd59d1c6701cfcb7f3c62b106: Status 404 returned error can't find the container with id 989f2222b482d1f28900b119a2a7f31684143f3dd59d1c6701cfcb7f3c62b106
Jul 31 01:51:13.592957 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:13.592929    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-l2mkg"]
Jul 31 01:51:13.595551 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:51:13.595522    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa01e1a2_7436_45ba_b058_346e7534e8ec.slice/crio-581243c92d457f6826ee8f92fe911204f3b7578a2ab7813305519c7fbf1d04f3 WatchSource:0}: Error finding container 581243c92d457f6826ee8f92fe911204f3b7578a2ab7813305519c7fbf1d04f3: Status 404 returned error can't find the container with id 581243c92d457f6826ee8f92fe911204f3b7578a2ab7813305519c7fbf1d04f3
Jul 31 01:51:14.084394 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:14.084353    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg" event={"ID":"fa01e1a2-7436-45ba-b058-346e7534e8ec","Type":"ContainerStarted","Data":"581243c92d457f6826ee8f92fe911204f3b7578a2ab7813305519c7fbf1d04f3"}
Jul 31 01:51:14.085330 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:14.085308    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9" event={"ID":"e66c5894-f7c2-4e5a-8a9c-ecd308c986df","Type":"ContainerStarted","Data":"989f2222b482d1f28900b119a2a7f31684143f3dd59d1c6701cfcb7f3c62b106"}
Jul 31 01:51:15.094009 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:15.093770    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj" event={"ID":"254c4573-cac4-45e1-9368-11a24bb2841e","Type":"ContainerStarted","Data":"f442d8397541a35229eb246597f5c5b578bbf4b5f5dac4323288def89ff0d19d"}
Jul 31 01:51:15.094009 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:15.093915    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:15.109889 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:15.109833    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj" podStartSLOduration=1.107345057 podStartE2EDuration="3.109817542s" totalImagesPullingTime="2.002472485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:51:15.108932638 +0000 UTC m=+833.474678281" watchObservedRunningTime="2026-07-31 01:51:15.109817542 +0000 UTC m=+833.475563167"
Jul 31 01:51:17.104018 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:17.103978    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9" event={"ID":"e66c5894-f7c2-4e5a-8a9c-ecd308c986df","Type":"ContainerStarted","Data":"39d1a67205a4268e741a67908e1235b4b6df9c8896d67c24b0f3ff6e506abe0c"}
Jul 31 01:51:17.104473 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:17.104069    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"
Jul 31 01:51:17.105402 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:17.105376    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg" event={"ID":"fa01e1a2-7436-45ba-b058-346e7534e8ec","Type":"ContainerStarted","Data":"97a2f757ba759fefe35869c2df8721bb1d19d46d6d683bb09e7c3dc5bee8fa65"}
Jul 31 01:51:17.105517 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:17.105504    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg"
Jul 31 01:51:17.119776 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:17.119734    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9" podStartSLOduration=1.573949058 podStartE2EDuration="4.119720868s" totalImagesPullingTime="2.54577181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:51:17.118749272 +0000 UTC m=+835.484494898" watchObservedRunningTime="2026-07-31 01:51:17.119720868 +0000 UTC m=+835.485466494"
Jul 31 01:51:17.134015 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:17.133977    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg" podStartSLOduration=1.608337849 podStartE2EDuration="4.133961287s" totalImagesPullingTime="2.525623438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:51:17.13347196 +0000 UTC m=+835.499217585" watchObservedRunningTime="2026-07-31 01:51:17.133961287 +0000 UTC m=+835.499706914"
Jul 31 01:51:21.103757 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:21.103727    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77f76cbbbd-27pnj"
Jul 31 01:51:22.819074 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:22.819034    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:51:22.925048 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:22.925013    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:51:28.112738 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:28.112711    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wfhq9"
Jul 31 01:51:28.113146 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:28.112916    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-l2mkg"
Jul 31 01:51:35.793141 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.793111    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"]
Jul 31 01:51:35.797508 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.797491    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.800622 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.800597    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 31 01:51:35.800724 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.800597    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 31 01:51:35.800778 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.800599    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 31 01:51:35.800778 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.800604    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4vc5j\""
Jul 31 01:51:35.804530 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.804509    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"]
Jul 31 01:51:35.809606 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.809584    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:51:35.871632 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.871592    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.871761 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.871645    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.871761 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.871682    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.871761 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.871736    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.871878 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.871774    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.871878 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.871838    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmm8g\" (UniqueName: \"kubernetes.io/projected/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-kube-api-access-tmm8g\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.973190 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.973162    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmm8g\" (UniqueName: \"kubernetes.io/projected/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-kube-api-access-tmm8g\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.973329 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.973198    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.973329 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.973237    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.973329 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.973269    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.973329 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.973287    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.973329 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.973319    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.973785 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.973764    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.973869 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.973802    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.973869 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.973826    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.975685 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.975662    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.975920 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.975903    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:35.980598 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:35.980549    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmm8g\" (UniqueName: \"kubernetes.io/projected/cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8-kube-api-access-tmm8g\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5\" (UID: \"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:36.107693 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:36.107670    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:36.239539 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:36.239475    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"]
Jul 31 01:51:36.242054 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:51:36.242022    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7bcc7e_32cc_47ff_b5e1_fd5000676ec8.slice/crio-f8a613d8e42f457891f4a152bf14be11b324e4648eeae937711c9d4d6de3ceb6 WatchSource:0}: Error finding container f8a613d8e42f457891f4a152bf14be11b324e4648eeae937711c9d4d6de3ceb6: Status 404 returned error can't find the container with id f8a613d8e42f457891f4a152bf14be11b324e4648eeae937711c9d4d6de3ceb6
Jul 31 01:51:37.187774 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:37.187733    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5" event={"ID":"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8","Type":"ContainerStarted","Data":"f8a613d8e42f457891f4a152bf14be11b324e4648eeae937711c9d4d6de3ceb6"}
Jul 31 01:51:38.193620 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:38.193583    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5" event={"ID":"cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8","Type":"ContainerStarted","Data":"74d1171340603ba41cfddd0812bcdb73737c449201d7158d0b921dc2b80b446d"}
Jul 31 01:51:38.209465 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:38.209417    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5" podStartSLOduration=2.148848396 podStartE2EDuration="3.209402355s" totalImagesPullingTime="1.060553959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:51:38.208397081 +0000 UTC m=+856.574142707" watchObservedRunningTime="2026-07-31 01:51:38.209402355 +0000 UTC m=+856.575147981"
Jul 31 01:51:46.107934 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:46.107904    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:46.108447 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:46.107984    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:46.124685 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:46.124655    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:46.243267 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:46.243237    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5"
Jul 31 01:51:52.833030 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:51:52.833000    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:52:14.824214 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:52:14.824137    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:52:25.227108 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:52:25.227067    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:52:42.818861 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:52:42.818823    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:52:49.511286 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:52:49.511250    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:53:34.807258 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:53:34.807200    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:53:34.904887 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:53:34.904841    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:53:35.913550 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:53:35.913511    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:53:36.013379 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:53:36.013343    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:53:52.892390 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:53:52.892359    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:54:05.845650 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:05.845616    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:54:05.894037 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:05.894008    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:54:22.235060 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.235024    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7745c54f68-sqzvv"]
Jul 31 01:54:22.235583 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.235213    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7745c54f68-sqzvv" podUID="63f0d476-d27a-4007-80da-df54354ca177" containerName="manager" containerID="cri-o://ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b" gracePeriod=30
Jul 31 01:54:22.482585 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.482545    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7745c54f68-sqzvv"
Jul 31 01:54:22.620297 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.620267    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63f0d476-d27a-4007-80da-df54354ca177-webhook-cert\") pod \"63f0d476-d27a-4007-80da-df54354ca177\" (UID: \"63f0d476-d27a-4007-80da-df54354ca177\") "
Jul 31 01:54:22.620473 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.620310    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cxnh\" (UniqueName: \"kubernetes.io/projected/63f0d476-d27a-4007-80da-df54354ca177-kube-api-access-7cxnh\") pod \"63f0d476-d27a-4007-80da-df54354ca177\" (UID: \"63f0d476-d27a-4007-80da-df54354ca177\") "
Jul 31 01:54:22.622492 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.622462    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f0d476-d27a-4007-80da-df54354ca177-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "63f0d476-d27a-4007-80da-df54354ca177" (UID: "63f0d476-d27a-4007-80da-df54354ca177"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 01:54:22.622612 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.622524    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f0d476-d27a-4007-80da-df54354ca177-kube-api-access-7cxnh" (OuterVolumeSpecName: "kube-api-access-7cxnh") pod "63f0d476-d27a-4007-80da-df54354ca177" (UID: "63f0d476-d27a-4007-80da-df54354ca177"). InnerVolumeSpecName "kube-api-access-7cxnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 01:54:22.721283 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.721248    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63f0d476-d27a-4007-80da-df54354ca177-webhook-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 01:54:22.721283 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.721279    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cxnh\" (UniqueName: \"kubernetes.io/projected/63f0d476-d27a-4007-80da-df54354ca177-kube-api-access-7cxnh\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 01:54:22.806682 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.806656    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:54:22.862760 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.862733    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:54:22.876083 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.876013    2556 generic.go:356] "Generic (PLEG): container finished" podID="63f0d476-d27a-4007-80da-df54354ca177" containerID="ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b" exitCode=0
Jul 31 01:54:22.876083 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.876053    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7745c54f68-sqzvv"
Jul 31 01:54:22.876230 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.876096    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7745c54f68-sqzvv" event={"ID":"63f0d476-d27a-4007-80da-df54354ca177","Type":"ContainerDied","Data":"ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b"}
Jul 31 01:54:22.876230 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.876141    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7745c54f68-sqzvv" event={"ID":"63f0d476-d27a-4007-80da-df54354ca177","Type":"ContainerDied","Data":"b9e8025e5cc44697dbce2ffaa8ad504d6c6602e882ca95b516c5bee2f530a24a"}
Jul 31 01:54:22.876230 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.876165    2556 scope.go:117] "RemoveContainer" containerID="ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b"
Jul 31 01:54:22.887011 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.886336    2556 scope.go:117] "RemoveContainer" containerID="ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b"
Jul 31 01:54:22.887369 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:54:22.887336    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b\": container with ID starting with ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b not found: ID does not exist" containerID="ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b"
Jul 31 01:54:22.887487 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.887380    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b"} err="failed to get container status \"ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b\": rpc error: code = NotFound desc = could not find container \"ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b\": container with ID starting with ffca8706044e6a88c79e10532538093364c0b36f42d01cd43e79bd39c6587c3b not found: ID does not exist"
Jul 31 01:54:22.896616 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.896587    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7745c54f68-sqzvv"]
Jul 31 01:54:22.900655 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:22.900635    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7745c54f68-sqzvv"]
Jul 31 01:54:24.151611 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:24.151576    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63f0d476-d27a-4007-80da-df54354ca177" path="/var/lib/kubelet/pods/63f0d476-d27a-4007-80da-df54354ca177/volumes"
Jul 31 01:54:54.858291 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:54.858252    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:54:54.907697 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:54:54.907668    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:55:15.560854 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:15.560779    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:55:23.726852 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.726815    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7745c54f68-5cn7s"]
Jul 31 01:55:23.727325 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.727282    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f0d476-d27a-4007-80da-df54354ca177" containerName="manager"
Jul 31 01:55:23.727325 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.727299    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f0d476-d27a-4007-80da-df54354ca177" containerName="manager"
Jul 31 01:55:23.727445 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.727384    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f0d476-d27a-4007-80da-df54354ca177" containerName="manager"
Jul 31 01:55:23.730586 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.730547    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7745c54f68-5cn7s"
Jul 31 01:55:23.733435 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.733413    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gwhc5\""
Jul 31 01:55:23.733523 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.733482    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 31 01:55:23.733523 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.733486    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 31 01:55:23.740183 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.740158    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7745c54f68-5cn7s"]
Jul 31 01:55:23.830678 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.830646    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzlq7\" (UniqueName: \"kubernetes.io/projected/f95a875b-b460-4f9f-88f9-f405ac55a6e1-kube-api-access-qzlq7\") pod \"maas-controller-7745c54f68-5cn7s\" (UID: \"f95a875b-b460-4f9f-88f9-f405ac55a6e1\") " pod="opendatahub/maas-controller-7745c54f68-5cn7s"
Jul 31 01:55:23.830817 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.830718    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f95a875b-b460-4f9f-88f9-f405ac55a6e1-webhook-cert\") pod \"maas-controller-7745c54f68-5cn7s\" (UID: \"f95a875b-b460-4f9f-88f9-f405ac55a6e1\") " pod="opendatahub/maas-controller-7745c54f68-5cn7s"
Jul 31 01:55:23.932014 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.931983    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzlq7\" (UniqueName: \"kubernetes.io/projected/f95a875b-b460-4f9f-88f9-f405ac55a6e1-kube-api-access-qzlq7\") pod \"maas-controller-7745c54f68-5cn7s\" (UID: \"f95a875b-b460-4f9f-88f9-f405ac55a6e1\") " pod="opendatahub/maas-controller-7745c54f68-5cn7s"
Jul 31 01:55:23.932170 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.932053    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f95a875b-b460-4f9f-88f9-f405ac55a6e1-webhook-cert\") pod \"maas-controller-7745c54f68-5cn7s\" (UID: \"f95a875b-b460-4f9f-88f9-f405ac55a6e1\") " pod="opendatahub/maas-controller-7745c54f68-5cn7s"
Jul 31 01:55:23.934400 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.934378    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f95a875b-b460-4f9f-88f9-f405ac55a6e1-webhook-cert\") pod \"maas-controller-7745c54f68-5cn7s\" (UID: \"f95a875b-b460-4f9f-88f9-f405ac55a6e1\") " pod="opendatahub/maas-controller-7745c54f68-5cn7s"
Jul 31 01:55:23.939514 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:23.939491    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzlq7\" (UniqueName: \"kubernetes.io/projected/f95a875b-b460-4f9f-88f9-f405ac55a6e1-kube-api-access-qzlq7\") pod \"maas-controller-7745c54f68-5cn7s\" (UID: \"f95a875b-b460-4f9f-88f9-f405ac55a6e1\") " pod="opendatahub/maas-controller-7745c54f68-5cn7s"
Jul 31 01:55:24.041886 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:24.041800    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7745c54f68-5cn7s"
Jul 31 01:55:24.170240 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:24.170114    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7745c54f68-5cn7s"]
Jul 31 01:55:24.172969 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:55:24.172929    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95a875b_b460_4f9f_88f9_f405ac55a6e1.slice/crio-e2fc6a1f97272dd80cf237ea6f2b18816337b9b8f07d77b9e2a1805bb5389038 WatchSource:0}: Error finding container e2fc6a1f97272dd80cf237ea6f2b18816337b9b8f07d77b9e2a1805bb5389038: Status 404 returned error can't find the container with id e2fc6a1f97272dd80cf237ea6f2b18816337b9b8f07d77b9e2a1805bb5389038
Jul 31 01:55:24.174090 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:24.174073    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 01:55:25.135691 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:25.135652    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7745c54f68-5cn7s" event={"ID":"f95a875b-b460-4f9f-88f9-f405ac55a6e1","Type":"ContainerStarted","Data":"e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197"}
Jul 31 01:55:25.136084 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:25.135741    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7745c54f68-5cn7s" event={"ID":"f95a875b-b460-4f9f-88f9-f405ac55a6e1","Type":"ContainerStarted","Data":"e2fc6a1f97272dd80cf237ea6f2b18816337b9b8f07d77b9e2a1805bb5389038"}
Jul 31 01:55:25.136084 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:25.135767    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7745c54f68-5cn7s"
Jul 31 01:55:25.152550 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:25.152495    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7745c54f68-5cn7s" podStartSLOduration=1.740564045 podStartE2EDuration="2.152479755s" totalImagesPullingTime="411.91571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:55:25.150632426 +0000 UTC m=+1083.516378056" watchObservedRunningTime="2026-07-31 01:55:25.152479755 +0000 UTC m=+1083.518225409"
Jul 31 01:55:36.142618 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:36.142589    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7745c54f68-5cn7s"
Jul 31 01:55:40.680881 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:40.680826    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:55:40.785623 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:40.785592    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7745c54f68-5cn7s"]
Jul 31 01:55:40.785818 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:40.785798    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7745c54f68-5cn7s" podUID="f95a875b-b460-4f9f-88f9-f405ac55a6e1" containerName="manager" containerID="cri-o://e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197" gracePeriod=30
Jul 31 01:55:41.026331 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.026309    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7745c54f68-5cn7s"
Jul 31 01:55:41.063535 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.063503    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f95a875b-b460-4f9f-88f9-f405ac55a6e1-webhook-cert\") pod \"f95a875b-b460-4f9f-88f9-f405ac55a6e1\" (UID: \"f95a875b-b460-4f9f-88f9-f405ac55a6e1\") "
Jul 31 01:55:41.063715 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.063546    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzlq7\" (UniqueName: \"kubernetes.io/projected/f95a875b-b460-4f9f-88f9-f405ac55a6e1-kube-api-access-qzlq7\") pod \"f95a875b-b460-4f9f-88f9-f405ac55a6e1\" (UID: \"f95a875b-b460-4f9f-88f9-f405ac55a6e1\") "
Jul 31 01:55:41.065735 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.065703    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95a875b-b460-4f9f-88f9-f405ac55a6e1-kube-api-access-qzlq7" (OuterVolumeSpecName: "kube-api-access-qzlq7") pod "f95a875b-b460-4f9f-88f9-f405ac55a6e1" (UID: "f95a875b-b460-4f9f-88f9-f405ac55a6e1"). InnerVolumeSpecName "kube-api-access-qzlq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 01:55:41.065857 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.065777    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f95a875b-b460-4f9f-88f9-f405ac55a6e1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f95a875b-b460-4f9f-88f9-f405ac55a6e1" (UID: "f95a875b-b460-4f9f-88f9-f405ac55a6e1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 01:55:41.164890 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.164864    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f95a875b-b460-4f9f-88f9-f405ac55a6e1-webhook-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 01:55:41.164890 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.164889    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzlq7\" (UniqueName: \"kubernetes.io/projected/f95a875b-b460-4f9f-88f9-f405ac55a6e1-kube-api-access-qzlq7\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 01:55:41.205360 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.205285    2556 generic.go:356] "Generic (PLEG): container finished" podID="f95a875b-b460-4f9f-88f9-f405ac55a6e1" containerID="e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197" exitCode=0
Jul 31 01:55:41.205360 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.205334    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7745c54f68-5cn7s"
Jul 31 01:55:41.205492 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.205382    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7745c54f68-5cn7s" event={"ID":"f95a875b-b460-4f9f-88f9-f405ac55a6e1","Type":"ContainerDied","Data":"e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197"}
Jul 31 01:55:41.205492 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.205430    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7745c54f68-5cn7s" event={"ID":"f95a875b-b460-4f9f-88f9-f405ac55a6e1","Type":"ContainerDied","Data":"e2fc6a1f97272dd80cf237ea6f2b18816337b9b8f07d77b9e2a1805bb5389038"}
Jul 31 01:55:41.205492 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.205450    2556 scope.go:117] "RemoveContainer" containerID="e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197"
Jul 31 01:55:41.214548 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.214532    2556 scope.go:117] "RemoveContainer" containerID="e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197"
Jul 31 01:55:41.214817 ip-10-0-141-142 kubenswrapper[2556]: E0731 01:55:41.214800    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197\": container with ID starting with e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197 not found: ID does not exist" containerID="e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197"
Jul 31 01:55:41.214857 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.214829    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197"} err="failed to get container status \"e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197\": rpc error: code = NotFound desc = could not find container \"e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197\": container with ID starting with e6efc33dd4d232c3eb0eee46b719713f723c08ed3cac4796d701ccd5fbfed197 not found: ID does not exist"
Jul 31 01:55:41.225783 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.225760    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7745c54f68-5cn7s"]
Jul 31 01:55:41.231784 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.231763    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7745c54f68-5cn7s"]
Jul 31 01:55:41.915988 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:41.915956    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:55:42.031480 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.031447    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7745c54f68-8lpz7"]
Jul 31 01:55:42.031774 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.031762    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f95a875b-b460-4f9f-88f9-f405ac55a6e1" containerName="manager"
Jul 31 01:55:42.031822 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.031775    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f95a875b-b460-4f9f-88f9-f405ac55a6e1" containerName="manager"
Jul 31 01:55:42.031855 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.031847    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f95a875b-b460-4f9f-88f9-f405ac55a6e1" containerName="manager"
Jul 31 01:55:42.035884 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.035868    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7745c54f68-8lpz7"
Jul 31 01:55:42.038604 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.038579    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 31 01:55:42.038727 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.038584    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gwhc5\""
Jul 31 01:55:42.038727 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.038616    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 31 01:55:42.044203 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.044179    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7745c54f68-8lpz7"]
Jul 31 01:55:42.070717 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.070686    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a44fce3b-698b-4e2d-8222-098849a2b9be-webhook-cert\") pod \"maas-controller-7745c54f68-8lpz7\" (UID: \"a44fce3b-698b-4e2d-8222-098849a2b9be\") " pod="opendatahub/maas-controller-7745c54f68-8lpz7"
Jul 31 01:55:42.070842 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.070769    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ld99\" (UniqueName: \"kubernetes.io/projected/a44fce3b-698b-4e2d-8222-098849a2b9be-kube-api-access-5ld99\") pod \"maas-controller-7745c54f68-8lpz7\" (UID: \"a44fce3b-698b-4e2d-8222-098849a2b9be\") " pod="opendatahub/maas-controller-7745c54f68-8lpz7"
Jul 31 01:55:42.150791 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.150764    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f95a875b-b460-4f9f-88f9-f405ac55a6e1" path="/var/lib/kubelet/pods/f95a875b-b460-4f9f-88f9-f405ac55a6e1/volumes"
Jul 31 01:55:42.172221 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.172171    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ld99\" (UniqueName: \"kubernetes.io/projected/a44fce3b-698b-4e2d-8222-098849a2b9be-kube-api-access-5ld99\") pod \"maas-controller-7745c54f68-8lpz7\" (UID: \"a44fce3b-698b-4e2d-8222-098849a2b9be\") " pod="opendatahub/maas-controller-7745c54f68-8lpz7"
Jul 31 01:55:42.172324 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.172227    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a44fce3b-698b-4e2d-8222-098849a2b9be-webhook-cert\") pod \"maas-controller-7745c54f68-8lpz7\" (UID: \"a44fce3b-698b-4e2d-8222-098849a2b9be\") " pod="opendatahub/maas-controller-7745c54f68-8lpz7"
Jul 31 01:55:42.174752 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.174727    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a44fce3b-698b-4e2d-8222-098849a2b9be-webhook-cert\") pod \"maas-controller-7745c54f68-8lpz7\" (UID: \"a44fce3b-698b-4e2d-8222-098849a2b9be\") " pod="opendatahub/maas-controller-7745c54f68-8lpz7"
Jul 31 01:55:42.179636 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.179610    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ld99\" (UniqueName: \"kubernetes.io/projected/a44fce3b-698b-4e2d-8222-098849a2b9be-kube-api-access-5ld99\") pod \"maas-controller-7745c54f68-8lpz7\" (UID: \"a44fce3b-698b-4e2d-8222-098849a2b9be\") " pod="opendatahub/maas-controller-7745c54f68-8lpz7"
Jul 31 01:55:42.274862 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.274833    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:55:42.347251 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.347223    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7745c54f68-8lpz7"
Jul 31 01:55:42.473659 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:42.473625    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7745c54f68-8lpz7"]
Jul 31 01:55:42.476678 ip-10-0-141-142 kubenswrapper[2556]: W0731 01:55:42.476649    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44fce3b_698b_4e2d_8222_098849a2b9be.slice/crio-0f6f967e9d8daa86e093fde1410dbcf3aef3f52c2b073565e7e3155bb4c8fa9c WatchSource:0}: Error finding container 0f6f967e9d8daa86e093fde1410dbcf3aef3f52c2b073565e7e3155bb4c8fa9c: Status 404 returned error can't find the container with id 0f6f967e9d8daa86e093fde1410dbcf3aef3f52c2b073565e7e3155bb4c8fa9c
Jul 31 01:55:43.221203 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:43.221121    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7745c54f68-8lpz7" event={"ID":"a44fce3b-698b-4e2d-8222-098849a2b9be","Type":"ContainerStarted","Data":"3298dd5a169dea1ab373a60e326f37da6202cdcd9f0eb21eee3f7f5d3f58b681"}
Jul 31 01:55:43.221203 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:43.221177    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7745c54f68-8lpz7" event={"ID":"a44fce3b-698b-4e2d-8222-098849a2b9be","Type":"ContainerStarted","Data":"0f6f967e9d8daa86e093fde1410dbcf3aef3f52c2b073565e7e3155bb4c8fa9c"}
Jul 31 01:55:43.221694 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:43.221230    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7745c54f68-8lpz7"
Jul 31 01:55:43.236842 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:43.236798    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7745c54f68-8lpz7" podStartSLOduration=0.904148126 podStartE2EDuration="1.23678568s" totalImagesPullingTime="332.637554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:55:43.235549891 +0000 UTC m=+1101.601295533" watchObservedRunningTime="2026-07-31 01:55:43.23678568 +0000 UTC m=+1101.602531308"
Jul 31 01:55:54.227788 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:55:54.227760    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7745c54f68-8lpz7"
Jul 31 01:56:41.867574 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:56:41.867530    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:56:41.964179 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:56:41.964144    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:56:42.066744 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:56:42.066714    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:56:47.927773 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:56:47.927701    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:56:47.982102 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:56:47.982068    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:56:48.061145 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:56:48.061111    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:56:48.160371 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:56:48.160292    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:56:48.284492 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:56:48.284394    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:56:54.084945 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:56:54.084908    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:56:54.184787 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:56:54.184755    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 01:59:57.148790 ip-10-0-141-142 kubenswrapper[2556]: I0731 01:59:57.148710    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:00:00.137939 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:00.137909    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757720-g6gfm"]
Jul 31 02:00:00.140948 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:00.140932    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757720-g6gfm"
Jul 31 02:00:00.143056 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:00.143032    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:00:00.150671 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:00.150654    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757720-g6gfm"
Jul 31 02:00:00.155065 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:00.155040    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757720-g6gfm"]
Jul 31 02:00:00.281915 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:00.281884    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757720-g6gfm"]
Jul 31 02:00:00.285017 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:00:00.284992    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9384ff_bddc_484e_916d_1745b71611d1.slice/crio-7db0e4694e9f92ffdceb59fa3f64a865624fe93684aa7143e53867de92362483 WatchSource:0}: Error finding container 7db0e4694e9f92ffdceb59fa3f64a865624fe93684aa7143e53867de92362483: Status 404 returned error can't find the container with id 7db0e4694e9f92ffdceb59fa3f64a865624fe93684aa7143e53867de92362483
Jul 31 02:00:01.277616 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:01.277546    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757720-g6gfm" event={"ID":"ba9384ff-bddc-484e-916d-1745b71611d1","Type":"ContainerStarted","Data":"7db0e4694e9f92ffdceb59fa3f64a865624fe93684aa7143e53867de92362483"}
Jul 31 02:00:03.288408 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:03.288375    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757720-g6gfm" event={"ID":"ba9384ff-bddc-484e-916d-1745b71611d1","Type":"ContainerStarted","Data":"62f19824b1a3fee281b242775a20af387d3496a0f6fece9152d36519f00d1e9b"}
Jul 31 02:00:03.302916 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:03.302873    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757720-g6gfm" podStartSLOduration=0.824864968 podStartE2EDuration="3.302858269s" totalImagesPullingTime="2.477993301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:00:03.301607499 +0000 UTC m=+1361.667353126" watchObservedRunningTime="2026-07-31 02:00:03.302858269 +0000 UTC m=+1361.668603897"
Jul 31 02:00:04.294051 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:04.294020    2556 generic.go:356] "Generic (PLEG): container finished" podID="ba9384ff-bddc-484e-916d-1745b71611d1" containerID="62f19824b1a3fee281b242775a20af387d3496a0f6fece9152d36519f00d1e9b" exitCode=0
Jul 31 02:00:04.294431 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:04.294092    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757720-g6gfm" event={"ID":"ba9384ff-bddc-484e-916d-1745b71611d1","Type":"ContainerDied","Data":"62f19824b1a3fee281b242775a20af387d3496a0f6fece9152d36519f00d1e9b"}
Jul 31 02:00:05.428114 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:05.428091    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757720-g6gfm"
Jul 31 02:00:06.304302 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:06.304277    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757720-g6gfm"
Jul 31 02:00:06.304458 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:06.304271    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757720-g6gfm" event={"ID":"ba9384ff-bddc-484e-916d-1745b71611d1","Type":"ContainerDied","Data":"7db0e4694e9f92ffdceb59fa3f64a865624fe93684aa7143e53867de92362483"}
Jul 31 02:00:06.304458 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:00:06.304406    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7db0e4694e9f92ffdceb59fa3f64a865624fe93684aa7143e53867de92362483"
Jul 31 02:03:46.909428 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:03:46.909383    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:03:46.998049 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:03:46.997993    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:04:01.822094 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:04:01.822066    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:04:02.383500 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:04:02.383462    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:05:00.205092 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:05:00.205056    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:06:13.326291 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:06:13.326199    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:06:25.186850 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:06:25.186788    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:07:08.869730 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:07:08.869687    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:07:10.515867 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:07:10.515820    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:10:10.149711 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:10:10.149676    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:10:11.446095 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:10:11.446062    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:10:32.192449 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:10:32.192413    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:10:58.175668 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:10:58.175632    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:01.192040 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:01.192004    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:01.292100 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:01.292067    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:01.640618 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:01.640587    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:07.389443 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:07.389412    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:07.489737 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:07.489710    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:07.637734 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:07.637713    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:07.951243 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:07.951211    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:08.061707 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:08.061675    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:10.316690 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:10.316653    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:10.408846 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:10.408817    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:11.524462 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.524432    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"]
Jul 31 02:11:11.524847 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.524755    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba9384ff-bddc-484e-916d-1745b71611d1" containerName="cleanup"
Jul 31 02:11:11.524847 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.524766    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba9384ff-bddc-484e-916d-1745b71611d1" containerName="cleanup"
Jul 31 02:11:11.524847 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.524826    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba9384ff-bddc-484e-916d-1745b71611d1" containerName="cleanup"
Jul 31 02:11:11.526728 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.526713    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:11.528912 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.528893    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5d8e59b6-serving-cert\""
Jul 31 02:11:11.538132 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.538110    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"]
Jul 31 02:11:11.563341 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.563320    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:11.587373 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.587343    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464a2e66-2413-44b1-be45-afe4ed76a5f9-openshift-service-ca\") pod \"maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:11.587481 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.587380    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464a2e66-2413-44b1-be45-afe4ed76a5f9-maas-api-tls\") pod \"maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:11.587481 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.587475    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw2g8\" (UniqueName: \"kubernetes.io/projected/464a2e66-2413-44b1-be45-afe4ed76a5f9-kube-api-access-dw2g8\") pod \"maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:11.688775 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.688748    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464a2e66-2413-44b1-be45-afe4ed76a5f9-openshift-service-ca\") pod \"maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:11.688775 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.688777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464a2e66-2413-44b1-be45-afe4ed76a5f9-maas-api-tls\") pod \"maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:11.688912 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.688813    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw2g8\" (UniqueName: \"kubernetes.io/projected/464a2e66-2413-44b1-be45-afe4ed76a5f9-kube-api-access-dw2g8\") pod \"maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:11.688912 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:11:11.688895    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-serving-cert: secret "maas-api-e2e-ait-5d8e59b6-serving-cert" not found
Jul 31 02:11:11.689011 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:11:11.688955    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/464a2e66-2413-44b1-be45-afe4ed76a5f9-maas-api-tls podName:464a2e66-2413-44b1-be45-afe4ed76a5f9 nodeName:}" failed. No retries permitted until 2026-07-31 02:11:12.188939988 +0000 UTC m=+2030.554685593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/464a2e66-2413-44b1-be45-afe4ed76a5f9-maas-api-tls") pod "maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p" (UID: "464a2e66-2413-44b1-be45-afe4ed76a5f9") : secret "maas-api-e2e-ait-5d8e59b6-serving-cert" not found
Jul 31 02:11:11.689335 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.689318    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464a2e66-2413-44b1-be45-afe4ed76a5f9-openshift-service-ca\") pod \"maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:11.696888 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:11.696867    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw2g8\" (UniqueName: \"kubernetes.io/projected/464a2e66-2413-44b1-be45-afe4ed76a5f9-kube-api-access-dw2g8\") pod \"maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:12.006000 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.005956    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:12.089961 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.089931    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:12.094274 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.094250    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"]
Jul 31 02:11:12.096482 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.096465    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"
Jul 31 02:11:12.098635 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.098616    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5d8e59b6\""
Jul 31 02:11:12.098724 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.098669    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5d8e59b6-dockercfg-fm2kj\""
Jul 31 02:11:12.107668 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.107648    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"]
Jul 31 02:11:12.134724 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.134705    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"]
Jul 31 02:11:12.136823 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.136809    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"
Jul 31 02:11:12.151585 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.151540    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"]
Jul 31 02:11:12.158577 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.158547    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:12.192017 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.191998    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464a2e66-2413-44b1-be45-afe4ed76a5f9-maas-api-tls\") pod \"maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:12.192108 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.192038    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knf4g\" (UniqueName: \"kubernetes.io/projected/647abed8-3606-41f7-a78f-648c864e1db4-kube-api-access-knf4g\") pod \"payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v\" (UID: \"647abed8-3606-41f7-a78f-648c864e1db4\") " pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"
Jul 31 02:11:12.192108 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.192061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df2460f-2d37-41eb-983b-102d36eef6da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw\" (UID: \"6df2460f-2d37-41eb-983b-102d36eef6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"
Jul 31 02:11:12.192108 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.192086    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647abed8-3606-41f7-a78f-648c864e1db4-plugins-config-volume\") pod \"payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v\" (UID: \"647abed8-3606-41f7-a78f-648c864e1db4\") " pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"
Jul 31 02:11:12.192234 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.192165    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c96h9\" (UniqueName: \"kubernetes.io/projected/6df2460f-2d37-41eb-983b-102d36eef6da-kube-api-access-c96h9\") pod \"payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw\" (UID: \"6df2460f-2d37-41eb-983b-102d36eef6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"
Jul 31 02:11:12.194401 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.194384    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464a2e66-2413-44b1-be45-afe4ed76a5f9-maas-api-tls\") pod \"maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:12.293054 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.292981    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c96h9\" (UniqueName: \"kubernetes.io/projected/6df2460f-2d37-41eb-983b-102d36eef6da-kube-api-access-c96h9\") pod \"payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw\" (UID: \"6df2460f-2d37-41eb-983b-102d36eef6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"
Jul 31 02:11:12.293054 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.293047    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knf4g\" (UniqueName: \"kubernetes.io/projected/647abed8-3606-41f7-a78f-648c864e1db4-kube-api-access-knf4g\") pod \"payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v\" (UID: \"647abed8-3606-41f7-a78f-648c864e1db4\") " pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"
Jul 31 02:11:12.293265 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.293064    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df2460f-2d37-41eb-983b-102d36eef6da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw\" (UID: \"6df2460f-2d37-41eb-983b-102d36eef6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"
Jul 31 02:11:12.293265 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.293083    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647abed8-3606-41f7-a78f-648c864e1db4-plugins-config-volume\") pod \"payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v\" (UID: \"647abed8-3606-41f7-a78f-648c864e1db4\") " pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"
Jul 31 02:11:12.293713 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.293694    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647abed8-3606-41f7-a78f-648c864e1db4-plugins-config-volume\") pod \"payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v\" (UID: \"647abed8-3606-41f7-a78f-648c864e1db4\") " pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"
Jul 31 02:11:12.293823 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.293806    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df2460f-2d37-41eb-983b-102d36eef6da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw\" (UID: \"6df2460f-2d37-41eb-983b-102d36eef6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"
Jul 31 02:11:12.302149 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.302122    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knf4g\" (UniqueName: \"kubernetes.io/projected/647abed8-3606-41f7-a78f-648c864e1db4-kube-api-access-knf4g\") pod \"payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v\" (UID: \"647abed8-3606-41f7-a78f-648c864e1db4\") " pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"
Jul 31 02:11:12.302248 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.302222    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c96h9\" (UniqueName: \"kubernetes.io/projected/6df2460f-2d37-41eb-983b-102d36eef6da-kube-api-access-c96h9\") pod \"payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw\" (UID: \"6df2460f-2d37-41eb-983b-102d36eef6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"
Jul 31 02:11:12.407642 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.407615    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"
Jul 31 02:11:12.437920 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.437897    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:12.445840 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.445819    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"
Jul 31 02:11:12.568386 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.568236    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"]
Jul 31 02:11:12.572833 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:11:12.572779    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df2460f_2d37_41eb_983b_102d36eef6da.slice/crio-713f79a9418145c1fe770770444ca4840b4ae75cd9d9872ebce3b1737e2bcfc7 WatchSource:0}: Error finding container 713f79a9418145c1fe770770444ca4840b4ae75cd9d9872ebce3b1737e2bcfc7: Status 404 returned error can't find the container with id 713f79a9418145c1fe770770444ca4840b4ae75cd9d9872ebce3b1737e2bcfc7
Jul 31 02:11:12.601200 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.601173    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"]
Jul 31 02:11:12.604071 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:11:12.604038    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464a2e66_2413_44b1_be45_afe4ed76a5f9.slice/crio-fb6d38d2bb62880b033d8b339ec2bc69ce92e6302b58d4c334ba1c41de38df50 WatchSource:0}: Error finding container fb6d38d2bb62880b033d8b339ec2bc69ce92e6302b58d4c334ba1c41de38df50: Status 404 returned error can't find the container with id fb6d38d2bb62880b033d8b339ec2bc69ce92e6302b58d4c334ba1c41de38df50
Jul 31 02:11:12.605364 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.605348    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 02:11:12.623464 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:12.623444    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"]
Jul 31 02:11:12.626355 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:11:12.626328    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647abed8_3606_41f7_a78f_648c864e1db4.slice/crio-6d265e63c89d4c6c4d1b8f0c4e2a0ed958766da6bd8d4eef41c31c608e31276d WatchSource:0}: Error finding container 6d265e63c89d4c6c4d1b8f0c4e2a0ed958766da6bd8d4eef41c31c608e31276d: Status 404 returned error can't find the container with id 6d265e63c89d4c6c4d1b8f0c4e2a0ed958766da6bd8d4eef41c31c608e31276d
Jul 31 02:11:13.056847 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:13.056803    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p" event={"ID":"464a2e66-2413-44b1-be45-afe4ed76a5f9","Type":"ContainerStarted","Data":"fb6d38d2bb62880b033d8b339ec2bc69ce92e6302b58d4c334ba1c41de38df50"}
Jul 31 02:11:13.058276 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:13.058246    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw" event={"ID":"6df2460f-2d37-41eb-983b-102d36eef6da","Type":"ContainerStarted","Data":"34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a"}
Jul 31 02:11:13.058396 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:13.058302    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw" event={"ID":"6df2460f-2d37-41eb-983b-102d36eef6da","Type":"ContainerStarted","Data":"713f79a9418145c1fe770770444ca4840b4ae75cd9d9872ebce3b1737e2bcfc7"}
Jul 31 02:11:13.058497 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:13.058471    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"
Jul 31 02:11:13.059831 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:13.059807    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v" event={"ID":"647abed8-3606-41f7-a78f-648c864e1db4","Type":"ContainerStarted","Data":"56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4"}
Jul 31 02:11:13.059934 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:13.059849    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v" event={"ID":"647abed8-3606-41f7-a78f-648c864e1db4","Type":"ContainerStarted","Data":"6d265e63c89d4c6c4d1b8f0c4e2a0ed958766da6bd8d4eef41c31c608e31276d"}
Jul 31 02:11:13.059934 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:13.059916    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"
Jul 31 02:11:13.076029 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:13.075980    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw" podStartSLOduration=1.075964112 podStartE2EDuration="1.075964112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:11:13.074320979 +0000 UTC m=+2031.440066606" watchObservedRunningTime="2026-07-31 02:11:13.075964112 +0000 UTC m=+2031.441709742"
Jul 31 02:11:13.089438 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:13.089389    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v" podStartSLOduration=1.089373715 podStartE2EDuration="1.089373715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:11:13.087713018 +0000 UTC m=+2031.453458647" watchObservedRunningTime="2026-07-31 02:11:13.089373715 +0000 UTC m=+2031.455119342"
Jul 31 02:11:14.065650 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:14.065608    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p" event={"ID":"464a2e66-2413-44b1-be45-afe4ed76a5f9","Type":"ContainerStarted","Data":"8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9"}
Jul 31 02:11:14.082260 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:14.082215    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p" podStartSLOduration=2.400056736 podStartE2EDuration="3.082202154s" totalImagesPullingTime="682.145418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:11:14.080889753 +0000 UTC m=+2032.446635382" watchObservedRunningTime="2026-07-31 02:11:14.082202154 +0000 UTC m=+2032.447947849"
Jul 31 02:11:15.072023 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:15.071989    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:21.081723 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:21.081693    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:24.068069 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:24.068044    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"
Jul 31 02:11:24.068428 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:24.068102    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"
Jul 31 02:11:35.925759 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:35.925725    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd"]
Jul 31 02:11:35.929018 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:35.928997    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd"
Jul 31 02:11:35.931026 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:35.930999    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:11:35.935822 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:35.935800    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd"]
Jul 31 02:11:36.060628 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:36.060600    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d37cee-f28c-4a1b-807c-27b7e2784305-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd\" (UID: \"20d37cee-f28c-4a1b-807c-27b7e2784305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd"
Jul 31 02:11:36.161680 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:36.161658    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d37cee-f28c-4a1b-807c-27b7e2784305-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd\" (UID: \"20d37cee-f28c-4a1b-807c-27b7e2784305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd"
Jul 31 02:11:36.162233 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:36.162213    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d37cee-f28c-4a1b-807c-27b7e2784305-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd\" (UID: \"20d37cee-f28c-4a1b-807c-27b7e2784305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd"
Jul 31 02:11:36.242448 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:36.242397    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd"
Jul 31 02:11:36.377463 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:36.377429    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd"]
Jul 31 02:11:36.380060 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:11:36.380026    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d37cee_f28c_4a1b_807c_27b7e2784305.slice/crio-bea649f29d10cc55bad2837d65a42caffd9a20c1d87cc8445d9c6fc3f0ad3cff WatchSource:0}: Error finding container bea649f29d10cc55bad2837d65a42caffd9a20c1d87cc8445d9c6fc3f0ad3cff: Status 404 returned error can't find the container with id bea649f29d10cc55bad2837d65a42caffd9a20c1d87cc8445d9c6fc3f0ad3cff
Jul 31 02:11:37.171806 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:37.171777    2556 generic.go:356] "Generic (PLEG): container finished" podID="20d37cee-f28c-4a1b-807c-27b7e2784305" containerID="ece540d92205e277c4fd1deadb7eaccd4c95053b2b8ced12feb82e11b358c311" exitCode=0
Jul 31 02:11:37.172112 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:37.171863    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd" event={"ID":"20d37cee-f28c-4a1b-807c-27b7e2784305","Type":"ContainerDied","Data":"ece540d92205e277c4fd1deadb7eaccd4c95053b2b8ced12feb82e11b358c311"}
Jul 31 02:11:37.172112 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:37.171916    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd" event={"ID":"20d37cee-f28c-4a1b-807c-27b7e2784305","Type":"ContainerStarted","Data":"bea649f29d10cc55bad2837d65a42caffd9a20c1d87cc8445d9c6fc3f0ad3cff"}
Jul 31 02:11:38.305166 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:38.305139    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd"
Jul 31 02:11:38.480527 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:38.480458    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d37cee-f28c-4a1b-807c-27b7e2784305-maas-api-service-ca\") pod \"20d37cee-f28c-4a1b-807c-27b7e2784305\" (UID: \"20d37cee-f28c-4a1b-807c-27b7e2784305\") "
Jul 31 02:11:38.480814 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:38.480790    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d37cee-f28c-4a1b-807c-27b7e2784305-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "20d37cee-f28c-4a1b-807c-27b7e2784305" (UID: "20d37cee-f28c-4a1b-807c-27b7e2784305"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:11:38.581065 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:38.581042    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d37cee-f28c-4a1b-807c-27b7e2784305-maas-api-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:11:39.181153 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:39.181124    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd" event={"ID":"20d37cee-f28c-4a1b-807c-27b7e2784305","Type":"ContainerDied","Data":"bea649f29d10cc55bad2837d65a42caffd9a20c1d87cc8445d9c6fc3f0ad3cff"}
Jul 31 02:11:39.181288 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:39.181156    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd"
Jul 31 02:11:39.181288 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:39.181164    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bea649f29d10cc55bad2837d65a42caffd9a20c1d87cc8445d9c6fc3f0ad3cff"
Jul 31 02:11:42.089551 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.089524    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"]
Jul 31 02:11:42.089965 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.089735    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p" podUID="464a2e66-2413-44b1-be45-afe4ed76a5f9" containerName="maas-api" containerID="cri-o://8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9" gracePeriod=30
Jul 31 02:11:42.120050 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.120024    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:42.142987 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.142963    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"]
Jul 31 02:11:42.143217 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.143193    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v" podUID="647abed8-3606-41f7-a78f-648c864e1db4" containerName="payload-processing" containerID="cri-o://56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4" gracePeriod=30
Jul 31 02:11:42.154232 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.154212    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"]
Jul 31 02:11:42.154391 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.154373    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw" podUID="6df2460f-2d37-41eb-983b-102d36eef6da" containerName="payload-pre-processing" containerID="cri-o://34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a" gracePeriod=30
Jul 31 02:11:42.220231 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.220205    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:42.271107 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.271078    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd"]
Jul 31 02:11:42.272781 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.272756    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d8e59b6-7zskd"]
Jul 31 02:11:42.276355 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.276333    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:42.427232 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.427203    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:42.439789 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.439763    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"
Jul 31 02:11:42.444445 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.444417    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:42.447517 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.447501    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"
Jul 31 02:11:42.610790 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.610723    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knf4g\" (UniqueName: \"kubernetes.io/projected/647abed8-3606-41f7-a78f-648c864e1db4-kube-api-access-knf4g\") pod \"647abed8-3606-41f7-a78f-648c864e1db4\" (UID: \"647abed8-3606-41f7-a78f-648c864e1db4\") "
Jul 31 02:11:42.610790 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.610766    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df2460f-2d37-41eb-983b-102d36eef6da-plugins-config-volume\") pod \"6df2460f-2d37-41eb-983b-102d36eef6da\" (UID: \"6df2460f-2d37-41eb-983b-102d36eef6da\") "
Jul 31 02:11:42.610790 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.610789    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464a2e66-2413-44b1-be45-afe4ed76a5f9-maas-api-tls\") pod \"464a2e66-2413-44b1-be45-afe4ed76a5f9\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") "
Jul 31 02:11:42.611077 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.610807    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c96h9\" (UniqueName: \"kubernetes.io/projected/6df2460f-2d37-41eb-983b-102d36eef6da-kube-api-access-c96h9\") pod \"6df2460f-2d37-41eb-983b-102d36eef6da\" (UID: \"6df2460f-2d37-41eb-983b-102d36eef6da\") "
Jul 31 02:11:42.611077 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.610830    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647abed8-3606-41f7-a78f-648c864e1db4-plugins-config-volume\") pod \"647abed8-3606-41f7-a78f-648c864e1db4\" (UID: \"647abed8-3606-41f7-a78f-648c864e1db4\") "
Jul 31 02:11:42.611077 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.610895    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw2g8\" (UniqueName: \"kubernetes.io/projected/464a2e66-2413-44b1-be45-afe4ed76a5f9-kube-api-access-dw2g8\") pod \"464a2e66-2413-44b1-be45-afe4ed76a5f9\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") "
Jul 31 02:11:42.611077 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.610937    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464a2e66-2413-44b1-be45-afe4ed76a5f9-openshift-service-ca\") pod \"464a2e66-2413-44b1-be45-afe4ed76a5f9\" (UID: \"464a2e66-2413-44b1-be45-afe4ed76a5f9\") "
Jul 31 02:11:42.611270 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.611183    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df2460f-2d37-41eb-983b-102d36eef6da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6df2460f-2d37-41eb-983b-102d36eef6da" (UID: "6df2460f-2d37-41eb-983b-102d36eef6da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:11:42.611270 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.611227    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647abed8-3606-41f7-a78f-648c864e1db4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "647abed8-3606-41f7-a78f-648c864e1db4" (UID: "647abed8-3606-41f7-a78f-648c864e1db4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:11:42.611585 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.611464    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464a2e66-2413-44b1-be45-afe4ed76a5f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "464a2e66-2413-44b1-be45-afe4ed76a5f9" (UID: "464a2e66-2413-44b1-be45-afe4ed76a5f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:11:42.613264 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.613234    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464a2e66-2413-44b1-be45-afe4ed76a5f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "464a2e66-2413-44b1-be45-afe4ed76a5f9" (UID: "464a2e66-2413-44b1-be45-afe4ed76a5f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:11:42.613264 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.613251    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df2460f-2d37-41eb-983b-102d36eef6da-kube-api-access-c96h9" (OuterVolumeSpecName: "kube-api-access-c96h9") pod "6df2460f-2d37-41eb-983b-102d36eef6da" (UID: "6df2460f-2d37-41eb-983b-102d36eef6da"). InnerVolumeSpecName "kube-api-access-c96h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:11:42.613400 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.613303    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647abed8-3606-41f7-a78f-648c864e1db4-kube-api-access-knf4g" (OuterVolumeSpecName: "kube-api-access-knf4g") pod "647abed8-3606-41f7-a78f-648c864e1db4" (UID: "647abed8-3606-41f7-a78f-648c864e1db4"). InnerVolumeSpecName "kube-api-access-knf4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:11:42.613400 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.613304    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464a2e66-2413-44b1-be45-afe4ed76a5f9-kube-api-access-dw2g8" (OuterVolumeSpecName: "kube-api-access-dw2g8") pod "464a2e66-2413-44b1-be45-afe4ed76a5f9" (UID: "464a2e66-2413-44b1-be45-afe4ed76a5f9"). InnerVolumeSpecName "kube-api-access-dw2g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:11:42.649704 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.649663    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:42.711725 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.711696    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knf4g\" (UniqueName: \"kubernetes.io/projected/647abed8-3606-41f7-a78f-648c864e1db4-kube-api-access-knf4g\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:11:42.711725 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.711721    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df2460f-2d37-41eb-983b-102d36eef6da-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:11:42.711877 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.711734    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464a2e66-2413-44b1-be45-afe4ed76a5f9-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:11:42.711877 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.711748    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c96h9\" (UniqueName: \"kubernetes.io/projected/6df2460f-2d37-41eb-983b-102d36eef6da-kube-api-access-c96h9\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:11:42.711877 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.711760    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647abed8-3606-41f7-a78f-648c864e1db4-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:11:42.711877 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.711772    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw2g8\" (UniqueName: \"kubernetes.io/projected/464a2e66-2413-44b1-be45-afe4ed76a5f9-kube-api-access-dw2g8\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:11:42.711877 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.711784    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464a2e66-2413-44b1-be45-afe4ed76a5f9-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:11:42.908583 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:42.908497    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:43.012947 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.012909    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:43.134616 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.134591    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:43.203638 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.203582    2556 generic.go:356] "Generic (PLEG): container finished" podID="464a2e66-2413-44b1-be45-afe4ed76a5f9" containerID="8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9" exitCode=0
Jul 31 02:11:43.203740 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.203637    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"
Jul 31 02:11:43.203740 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.203645    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p" event={"ID":"464a2e66-2413-44b1-be45-afe4ed76a5f9","Type":"ContainerDied","Data":"8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9"}
Jul 31 02:11:43.203740 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.203732    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p" event={"ID":"464a2e66-2413-44b1-be45-afe4ed76a5f9","Type":"ContainerDied","Data":"fb6d38d2bb62880b033d8b339ec2bc69ce92e6302b58d4c334ba1c41de38df50"}
Jul 31 02:11:43.203881 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.203760    2556 scope.go:117] "RemoveContainer" containerID="8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9"
Jul 31 02:11:43.205099 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.204997    2556 generic.go:356] "Generic (PLEG): container finished" podID="6df2460f-2d37-41eb-983b-102d36eef6da" containerID="34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a" exitCode=0
Jul 31 02:11:43.205099 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.205034    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"
Jul 31 02:11:43.205099 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.205062    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw" event={"ID":"6df2460f-2d37-41eb-983b-102d36eef6da","Type":"ContainerDied","Data":"34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a"}
Jul 31 02:11:43.205253 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.205101    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw" event={"ID":"6df2460f-2d37-41eb-983b-102d36eef6da","Type":"ContainerDied","Data":"713f79a9418145c1fe770770444ca4840b4ae75cd9d9872ebce3b1737e2bcfc7"}
Jul 31 02:11:43.206443 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.206423    2556 generic.go:356] "Generic (PLEG): container finished" podID="647abed8-3606-41f7-a78f-648c864e1db4" containerID="56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4" exitCode=0
Jul 31 02:11:43.206539 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.206463    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v" event={"ID":"647abed8-3606-41f7-a78f-648c864e1db4","Type":"ContainerDied","Data":"56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4"}
Jul 31 02:11:43.206539 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.206482    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"
Jul 31 02:11:43.206539 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.206510    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v" event={"ID":"647abed8-3606-41f7-a78f-648c864e1db4","Type":"ContainerDied","Data":"6d265e63c89d4c6c4d1b8f0c4e2a0ed958766da6bd8d4eef41c31c608e31276d"}
Jul 31 02:11:43.213891 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.213875    2556 scope.go:117] "RemoveContainer" containerID="8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9"
Jul 31 02:11:43.214170 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:11:43.214147    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9\": container with ID starting with 8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9 not found: ID does not exist" containerID="8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9"
Jul 31 02:11:43.214251 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.214180    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9"} err="failed to get container status \"8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9\": rpc error: code = NotFound desc = could not find container \"8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9\": container with ID starting with 8312e51a91f76f593568bcbcd0298f41f814e22ac845904ee0e3c26169104cc9 not found: ID does not exist"
Jul 31 02:11:43.214251 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.214200    2556 scope.go:117] "RemoveContainer" containerID="34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a"
Jul 31 02:11:43.221827 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.221810    2556 scope.go:117] "RemoveContainer" containerID="34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a"
Jul 31 02:11:43.222056 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:11:43.222040    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a\": container with ID starting with 34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a not found: ID does not exist" containerID="34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a"
Jul 31 02:11:43.222120 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.222063    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a"} err="failed to get container status \"34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a\": rpc error: code = NotFound desc = could not find container \"34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a\": container with ID starting with 34ab7f54ab8e0a7fbb4e1c89db1921a740a1eecdd2611f9be0badf3821c6f42a not found: ID does not exist"
Jul 31 02:11:43.222120 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.222083    2556 scope.go:117] "RemoveContainer" containerID="56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4"
Jul 31 02:11:43.230427 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.230381    2556 scope.go:117] "RemoveContainer" containerID="56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4"
Jul 31 02:11:43.230870 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:11:43.230834    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4\": container with ID starting with 56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4 not found: ID does not exist" containerID="56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4"
Jul 31 02:11:43.230997 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.230977    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4"} err="failed to get container status \"56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4\": rpc error: code = NotFound desc = could not find container \"56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4\": container with ID starting with 56c0b90154220b06b8fd73e2f68912802bcf165290aadd965a84555b75d734f4 not found: ID does not exist"
Jul 31 02:11:43.232604 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.232585    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"]
Jul 31 02:11:43.234908 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.234890    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d8e59b6-9d78445d4-pjdbw"]
Jul 31 02:11:43.244586 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.244550    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"]
Jul 31 02:11:43.246464 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.246445    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d8e59b6-767d86c695-l2f5p"]
Jul 31 02:11:43.255526 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.255506    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"]
Jul 31 02:11:43.259195 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.259175    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d8e59b6-59658df844-k6w5v"]
Jul 31 02:11:43.449411 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.449377    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:43.617766 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:43.617739    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:44.150291 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:44.150263    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20d37cee-f28c-4a1b-807c-27b7e2784305" path="/var/lib/kubelet/pods/20d37cee-f28c-4a1b-807c-27b7e2784305/volumes"
Jul 31 02:11:44.150646 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:44.150627    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="464a2e66-2413-44b1-be45-afe4ed76a5f9" path="/var/lib/kubelet/pods/464a2e66-2413-44b1-be45-afe4ed76a5f9/volumes"
Jul 31 02:11:44.150915 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:44.150903    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="647abed8-3606-41f7-a78f-648c864e1db4" path="/var/lib/kubelet/pods/647abed8-3606-41f7-a78f-648c864e1db4/volumes"
Jul 31 02:11:44.151224 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:44.151212    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df2460f-2d37-41eb-983b-102d36eef6da" path="/var/lib/kubelet/pods/6df2460f-2d37-41eb-983b-102d36eef6da/volumes"
Jul 31 02:11:45.621350 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.621315    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:45.692163 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692119    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"]
Jul 31 02:11:45.692441 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692430    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464a2e66-2413-44b1-be45-afe4ed76a5f9" containerName="maas-api"
Jul 31 02:11:45.692487 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692443    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="464a2e66-2413-44b1-be45-afe4ed76a5f9" containerName="maas-api"
Jul 31 02:11:45.692487 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692452    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d37cee-f28c-4a1b-807c-27b7e2784305" containerName="revoke-keys"
Jul 31 02:11:45.692487 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692457    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d37cee-f28c-4a1b-807c-27b7e2784305" containerName="revoke-keys"
Jul 31 02:11:45.692487 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692465    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="647abed8-3606-41f7-a78f-648c864e1db4" containerName="payload-processing"
Jul 31 02:11:45.692487 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692471    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="647abed8-3606-41f7-a78f-648c864e1db4" containerName="payload-processing"
Jul 31 02:11:45.692487 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692480    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df2460f-2d37-41eb-983b-102d36eef6da" containerName="payload-pre-processing"
Jul 31 02:11:45.692487 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692485    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df2460f-2d37-41eb-983b-102d36eef6da" containerName="payload-pre-processing"
Jul 31 02:11:45.692743 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692526    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df2460f-2d37-41eb-983b-102d36eef6da" containerName="payload-pre-processing"
Jul 31 02:11:45.692743 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692535    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d37cee-f28c-4a1b-807c-27b7e2784305" containerName="revoke-keys"
Jul 31 02:11:45.692743 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692541    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="647abed8-3606-41f7-a78f-648c864e1db4" containerName="payload-processing"
Jul 31 02:11:45.692743 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.692547    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="464a2e66-2413-44b1-be45-afe4ed76a5f9" containerName="maas-api"
Jul 31 02:11:45.697246 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.697228    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:45.713093 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.713077    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-241ce02f-serving-cert\""
Jul 31 02:11:45.724697 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.724672    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"]
Jul 31 02:11:45.778186 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.778165    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:45.832758 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.832725    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5509762-503f-4d3f-87d4-6959703ee42d-maas-api-tls\") pod \"maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9\" (UID: \"f5509762-503f-4d3f-87d4-6959703ee42d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:45.832758 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.832756    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5509762-503f-4d3f-87d4-6959703ee42d-openshift-service-ca\") pod \"maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9\" (UID: \"f5509762-503f-4d3f-87d4-6959703ee42d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:45.832905 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.832774    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfzdl\" (UniqueName: \"kubernetes.io/projected/f5509762-503f-4d3f-87d4-6959703ee42d-kube-api-access-hfzdl\") pod \"maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9\" (UID: \"f5509762-503f-4d3f-87d4-6959703ee42d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:45.933463 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.933400    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5509762-503f-4d3f-87d4-6959703ee42d-maas-api-tls\") pod \"maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9\" (UID: \"f5509762-503f-4d3f-87d4-6959703ee42d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:45.933463 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.933435    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5509762-503f-4d3f-87d4-6959703ee42d-openshift-service-ca\") pod \"maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9\" (UID: \"f5509762-503f-4d3f-87d4-6959703ee42d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:45.933463 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.933460    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfzdl\" (UniqueName: \"kubernetes.io/projected/f5509762-503f-4d3f-87d4-6959703ee42d-kube-api-access-hfzdl\") pod \"maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9\" (UID: \"f5509762-503f-4d3f-87d4-6959703ee42d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:45.934007 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.933988    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5509762-503f-4d3f-87d4-6959703ee42d-openshift-service-ca\") pod \"maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9\" (UID: \"f5509762-503f-4d3f-87d4-6959703ee42d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:45.935947 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.935924    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5509762-503f-4d3f-87d4-6959703ee42d-maas-api-tls\") pod \"maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9\" (UID: \"f5509762-503f-4d3f-87d4-6959703ee42d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:45.941135 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:45.941104    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfzdl\" (UniqueName: \"kubernetes.io/projected/f5509762-503f-4d3f-87d4-6959703ee42d-kube-api-access-hfzdl\") pod \"maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9\" (UID: \"f5509762-503f-4d3f-87d4-6959703ee42d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:46.007412 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.007393    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:46.128658 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.128599    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:46.151138 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.151120    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"]
Jul 31 02:11:46.153124 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:11:46.153100    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5509762_503f_4d3f_87d4_6959703ee42d.slice/crio-b2bf4f37248e2dc3796cb24f93acc04483df0e23c00e54767b051c4eee7bd88a WatchSource:0}: Error finding container b2bf4f37248e2dc3796cb24f93acc04483df0e23c00e54767b051c4eee7bd88a: Status 404 returned error can't find the container with id b2bf4f37248e2dc3796cb24f93acc04483df0e23c00e54767b051c4eee7bd88a
Jul 31 02:11:46.222298 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.222227    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9" event={"ID":"f5509762-503f-4d3f-87d4-6959703ee42d","Type":"ContainerStarted","Data":"b2bf4f37248e2dc3796cb24f93acc04483df0e23c00e54767b051c4eee7bd88a"}
Jul 31 02:11:46.271912 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.271882    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:46.359230 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.359196    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"]
Jul 31 02:11:46.361739 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.361702    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:11:46.363721 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.363702    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"
Jul 31 02:11:46.365806 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.365790    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-241ce02f\""
Jul 31 02:11:46.374023 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.374004    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"]
Jul 31 02:11:46.408281 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.408257    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"]
Jul 31 02:11:46.411342 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.411323    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"
Jul 31 02:11:46.413380 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.413364    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-241ce02f-dockercfg-xfpjh\""
Jul 31 02:11:46.421873 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.421853    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"]
Jul 31 02:11:46.539420 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.539399    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw75g\" (UniqueName: \"kubernetes.io/projected/f40bd54f-a2fe-409c-8396-c2707b94ac95-kube-api-access-cw75g\") pod \"payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p\" (UID: \"f40bd54f-a2fe-409c-8396-c2707b94ac95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"
Jul 31 02:11:46.539517 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.539438    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9236229-2cc7-4272-89ff-70d0082dddc9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2\" (UID: \"a9236229-2cc7-4272-89ff-70d0082dddc9\") " pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"
Jul 31 02:11:46.539517 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.539467    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40bd54f-a2fe-409c-8396-c2707b94ac95-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p\" (UID: \"f40bd54f-a2fe-409c-8396-c2707b94ac95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"
Jul 31 02:11:46.539633 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.539528    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bswd\" (UniqueName: \"kubernetes.io/projected/a9236229-2cc7-4272-89ff-70d0082dddc9-kube-api-access-9bswd\") pod \"payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2\" (UID: \"a9236229-2cc7-4272-89ff-70d0082dddc9\") " pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"
Jul 31 02:11:46.640184 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.640163    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw75g\" (UniqueName: \"kubernetes.io/projected/f40bd54f-a2fe-409c-8396-c2707b94ac95-kube-api-access-cw75g\") pod \"payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p\" (UID: \"f40bd54f-a2fe-409c-8396-c2707b94ac95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"
Jul 31 02:11:46.640534 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.640199    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9236229-2cc7-4272-89ff-70d0082dddc9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2\" (UID: \"a9236229-2cc7-4272-89ff-70d0082dddc9\") " pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"
Jul 31 02:11:46.640534 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.640226    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40bd54f-a2fe-409c-8396-c2707b94ac95-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p\" (UID: \"f40bd54f-a2fe-409c-8396-c2707b94ac95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"
Jul 31 02:11:46.640534 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.640247    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bswd\" (UniqueName: \"kubernetes.io/projected/a9236229-2cc7-4272-89ff-70d0082dddc9-kube-api-access-9bswd\") pod \"payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2\" (UID: \"a9236229-2cc7-4272-89ff-70d0082dddc9\") " pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"
Jul 31 02:11:46.641080 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.641058    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9236229-2cc7-4272-89ff-70d0082dddc9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2\" (UID: \"a9236229-2cc7-4272-89ff-70d0082dddc9\") " pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"
Jul 31 02:11:46.641114 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.641060    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40bd54f-a2fe-409c-8396-c2707b94ac95-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p\" (UID: \"f40bd54f-a2fe-409c-8396-c2707b94ac95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"
Jul 31 02:11:46.648503 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.648478    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw75g\" (UniqueName: \"kubernetes.io/projected/f40bd54f-a2fe-409c-8396-c2707b94ac95-kube-api-access-cw75g\") pod \"payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p\" (UID: \"f40bd54f-a2fe-409c-8396-c2707b94ac95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"
Jul 31 02:11:46.648901 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.648885    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bswd\" (UniqueName: \"kubernetes.io/projected/a9236229-2cc7-4272-89ff-70d0082dddc9-kube-api-access-9bswd\") pod \"payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2\" (UID: \"a9236229-2cc7-4272-89ff-70d0082dddc9\") " pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"
Jul 31 02:11:46.676232 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.676207    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"
Jul 31 02:11:46.721731 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.721622    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"
Jul 31 02:11:46.815999 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.815973    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"]
Jul 31 02:11:46.817261 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:11:46.817236    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40bd54f_a2fe_409c_8396_c2707b94ac95.slice/crio-491de0752faaafd99f11c0382a02598923d058cb1e4fe269c5f8676b0f583f7b WatchSource:0}: Error finding container 491de0752faaafd99f11c0382a02598923d058cb1e4fe269c5f8676b0f583f7b: Status 404 returned error can't find the container with id 491de0752faaafd99f11c0382a02598923d058cb1e4fe269c5f8676b0f583f7b
Jul 31 02:11:46.875711 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:46.875683    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"]
Jul 31 02:11:46.877119 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:11:46.877094    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9236229_2cc7_4272_89ff_70d0082dddc9.slice/crio-de465c5e67e8d1ab873bba490fdf5fcae934f57c2ac17e0fc7647afd0110249d WatchSource:0}: Error finding container de465c5e67e8d1ab873bba490fdf5fcae934f57c2ac17e0fc7647afd0110249d: Status 404 returned error can't find the container with id de465c5e67e8d1ab873bba490fdf5fcae934f57c2ac17e0fc7647afd0110249d
Jul 31 02:11:47.231290 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:47.231256    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9" event={"ID":"f5509762-503f-4d3f-87d4-6959703ee42d","Type":"ContainerStarted","Data":"8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6"}
Jul 31 02:11:47.231460 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:47.231324    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:47.232924 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:47.232894    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p" event={"ID":"f40bd54f-a2fe-409c-8396-c2707b94ac95","Type":"ContainerStarted","Data":"d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b"}
Jul 31 02:11:47.233039 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:47.232935    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p" event={"ID":"f40bd54f-a2fe-409c-8396-c2707b94ac95","Type":"ContainerStarted","Data":"491de0752faaafd99f11c0382a02598923d058cb1e4fe269c5f8676b0f583f7b"}
Jul 31 02:11:47.233039 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:47.232961    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"
Jul 31 02:11:47.234325 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:47.234299    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2" event={"ID":"a9236229-2cc7-4272-89ff-70d0082dddc9","Type":"ContainerStarted","Data":"f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94"}
Jul 31 02:11:47.234459 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:47.234351    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2" event={"ID":"a9236229-2cc7-4272-89ff-70d0082dddc9","Type":"ContainerStarted","Data":"de465c5e67e8d1ab873bba490fdf5fcae934f57c2ac17e0fc7647afd0110249d"}
Jul 31 02:11:47.234459 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:47.234386    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"
Jul 31 02:11:47.247017 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:47.246964    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9" podStartSLOduration=1.883094939 podStartE2EDuration="2.246947265s" totalImagesPullingTime="363.852326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:11:47.246081671 +0000 UTC m=+2065.611827309" watchObservedRunningTime="2026-07-31 02:11:47.246947265 +0000 UTC m=+2065.612692894"
Jul 31 02:11:47.260796 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:47.260762    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2" podStartSLOduration=1.26074968 podStartE2EDuration="1.26074968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:11:47.259516638 +0000 UTC m=+2065.625262267" watchObservedRunningTime="2026-07-31 02:11:47.26074968 +0000 UTC m=+2065.626495333"
Jul 31 02:11:47.274217 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:47.274181    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p" podStartSLOduration=1.27416475 podStartE2EDuration="1.27416475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:11:47.273200978 +0000 UTC m=+2065.638946707" watchObservedRunningTime="2026-07-31 02:11:47.27416475 +0000 UTC m=+2065.639910376"
Jul 31 02:11:53.243326 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:53.243298    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:11:58.240248 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:58.240206    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"
Jul 31 02:11:58.240659 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:11:58.240444    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"
Jul 31 02:12:10.628660 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:10.628626    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt"]
Jul 31 02:12:10.632184 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:10.632162    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt"
Jul 31 02:12:10.634231 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:10.634214    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:12:10.639492 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:10.639468    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt"]
Jul 31 02:12:10.719460 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:10.719432    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5f98ce-6758-4b66-b9f1-ae0b1db76675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-241ce02f-748vt\" (UID: \"5e5f98ce-6758-4b66-b9f1-ae0b1db76675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt"
Jul 31 02:12:10.819877 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:10.819852    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5f98ce-6758-4b66-b9f1-ae0b1db76675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-241ce02f-748vt\" (UID: \"5e5f98ce-6758-4b66-b9f1-ae0b1db76675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt"
Jul 31 02:12:10.820374 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:10.820357    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5f98ce-6758-4b66-b9f1-ae0b1db76675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-241ce02f-748vt\" (UID: \"5e5f98ce-6758-4b66-b9f1-ae0b1db76675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt"
Jul 31 02:12:10.944196 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:10.944118    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt"
Jul 31 02:12:11.096362 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:11.096338    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt"]
Jul 31 02:12:11.099233 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:12:11.099208    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5f98ce_6758_4b66_b9f1_ae0b1db76675.slice/crio-53c4773bf1acf17dfd8371fdb41ca2be8b63f03781a58370158859af5864bcea WatchSource:0}: Error finding container 53c4773bf1acf17dfd8371fdb41ca2be8b63f03781a58370158859af5864bcea: Status 404 returned error can't find the container with id 53c4773bf1acf17dfd8371fdb41ca2be8b63f03781a58370158859af5864bcea
Jul 31 02:12:11.337070 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:11.337043    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt" event={"ID":"5e5f98ce-6758-4b66-b9f1-ae0b1db76675","Type":"ContainerStarted","Data":"53c4773bf1acf17dfd8371fdb41ca2be8b63f03781a58370158859af5864bcea"}
Jul 31 02:12:12.342627 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:12.342593    2556 generic.go:356] "Generic (PLEG): container finished" podID="5e5f98ce-6758-4b66-b9f1-ae0b1db76675" containerID="afd9d035a510f57d28e1c3f9ddbfd1124f8121da60fc3edd658a578c246f9c0c" exitCode=0
Jul 31 02:12:12.342989 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:12.342674    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt" event={"ID":"5e5f98ce-6758-4b66-b9f1-ae0b1db76675","Type":"ContainerDied","Data":"afd9d035a510f57d28e1c3f9ddbfd1124f8121da60fc3edd658a578c246f9c0c"}
Jul 31 02:12:13.474490 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:13.474467    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt"
Jul 31 02:12:13.539261 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:13.539234    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5f98ce-6758-4b66-b9f1-ae0b1db76675-maas-api-service-ca\") pod \"5e5f98ce-6758-4b66-b9f1-ae0b1db76675\" (UID: \"5e5f98ce-6758-4b66-b9f1-ae0b1db76675\") "
Jul 31 02:12:13.539523 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:13.539503    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5f98ce-6758-4b66-b9f1-ae0b1db76675-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e5f98ce-6758-4b66-b9f1-ae0b1db76675" (UID: "5e5f98ce-6758-4b66-b9f1-ae0b1db76675"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:12:13.640613 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:13.640552    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5f98ce-6758-4b66-b9f1-ae0b1db76675-maas-api-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:14.353696 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:14.353678    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt"
Jul 31 02:12:14.353810 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:14.353690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt" event={"ID":"5e5f98ce-6758-4b66-b9f1-ae0b1db76675","Type":"ContainerDied","Data":"53c4773bf1acf17dfd8371fdb41ca2be8b63f03781a58370158859af5864bcea"}
Jul 31 02:12:14.353810 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:14.353731    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53c4773bf1acf17dfd8371fdb41ca2be8b63f03781a58370158859af5864bcea"
Jul 31 02:12:16.327525 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.327498    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"]
Jul 31 02:12:16.327963 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.327722    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9" podUID="f5509762-503f-4d3f-87d4-6959703ee42d" containerName="maas-api" containerID="cri-o://8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6" gracePeriod=30
Jul 31 02:12:16.353118 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.353071    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:16.382200 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.382173    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"]
Jul 31 02:12:16.382392 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.382365    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2" podUID="a9236229-2cc7-4272-89ff-70d0082dddc9" containerName="payload-processing" containerID="cri-o://f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94" gracePeriod=30
Jul 31 02:12:16.388108 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.388082    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"]
Jul 31 02:12:16.388323 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.388294    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p" podUID="f40bd54f-a2fe-409c-8396-c2707b94ac95" containerName="payload-pre-processing" containerID="cri-o://d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b" gracePeriod=30
Jul 31 02:12:16.418827 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.418765    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:16.477653 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.477623    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:16.512023 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.511980    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt"]
Jul 31 02:12:16.515307 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.515266    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-241ce02f-748vt"]
Jul 31 02:12:16.538243 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.537485    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:16.651719 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.651696    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:12:16.655347 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.655171    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"
Jul 31 02:12:16.663215 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.663189    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:16.696844 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.696820    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"
Jul 31 02:12:16.761931 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.761902    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5509762-503f-4d3f-87d4-6959703ee42d-openshift-service-ca\") pod \"f5509762-503f-4d3f-87d4-6959703ee42d\" (UID: \"f5509762-503f-4d3f-87d4-6959703ee42d\") "
Jul 31 02:12:16.762076 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.761947    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw75g\" (UniqueName: \"kubernetes.io/projected/f40bd54f-a2fe-409c-8396-c2707b94ac95-kube-api-access-cw75g\") pod \"f40bd54f-a2fe-409c-8396-c2707b94ac95\" (UID: \"f40bd54f-a2fe-409c-8396-c2707b94ac95\") "
Jul 31 02:12:16.762076 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.761986    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5509762-503f-4d3f-87d4-6959703ee42d-maas-api-tls\") pod \"f5509762-503f-4d3f-87d4-6959703ee42d\" (UID: \"f5509762-503f-4d3f-87d4-6959703ee42d\") "
Jul 31 02:12:16.762076 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.762027    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfzdl\" (UniqueName: \"kubernetes.io/projected/f5509762-503f-4d3f-87d4-6959703ee42d-kube-api-access-hfzdl\") pod \"f5509762-503f-4d3f-87d4-6959703ee42d\" (UID: \"f5509762-503f-4d3f-87d4-6959703ee42d\") "
Jul 31 02:12:16.762076 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.762048    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9236229-2cc7-4272-89ff-70d0082dddc9-plugins-config-volume\") pod \"a9236229-2cc7-4272-89ff-70d0082dddc9\" (UID: \"a9236229-2cc7-4272-89ff-70d0082dddc9\") "
Jul 31 02:12:16.762076 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.762065    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bswd\" (UniqueName: \"kubernetes.io/projected/a9236229-2cc7-4272-89ff-70d0082dddc9-kube-api-access-9bswd\") pod \"a9236229-2cc7-4272-89ff-70d0082dddc9\" (UID: \"a9236229-2cc7-4272-89ff-70d0082dddc9\") "
Jul 31 02:12:16.762329 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.762086    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40bd54f-a2fe-409c-8396-c2707b94ac95-plugins-config-volume\") pod \"f40bd54f-a2fe-409c-8396-c2707b94ac95\" (UID: \"f40bd54f-a2fe-409c-8396-c2707b94ac95\") "
Jul 31 02:12:16.762466 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.762375    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5509762-503f-4d3f-87d4-6959703ee42d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f5509762-503f-4d3f-87d4-6959703ee42d" (UID: "f5509762-503f-4d3f-87d4-6959703ee42d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:12:16.762584 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.762463    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9236229-2cc7-4272-89ff-70d0082dddc9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9236229-2cc7-4272-89ff-70d0082dddc9" (UID: "a9236229-2cc7-4272-89ff-70d0082dddc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:12:16.762644 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.762595    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40bd54f-a2fe-409c-8396-c2707b94ac95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f40bd54f-a2fe-409c-8396-c2707b94ac95" (UID: "f40bd54f-a2fe-409c-8396-c2707b94ac95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:12:16.764398 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.764376    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5509762-503f-4d3f-87d4-6959703ee42d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f5509762-503f-4d3f-87d4-6959703ee42d" (UID: "f5509762-503f-4d3f-87d4-6959703ee42d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:12:16.764487 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.764438    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9236229-2cc7-4272-89ff-70d0082dddc9-kube-api-access-9bswd" (OuterVolumeSpecName: "kube-api-access-9bswd") pod "a9236229-2cc7-4272-89ff-70d0082dddc9" (UID: "a9236229-2cc7-4272-89ff-70d0082dddc9"). InnerVolumeSpecName "kube-api-access-9bswd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:12:16.764734 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.764710    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5509762-503f-4d3f-87d4-6959703ee42d-kube-api-access-hfzdl" (OuterVolumeSpecName: "kube-api-access-hfzdl") pod "f5509762-503f-4d3f-87d4-6959703ee42d" (UID: "f5509762-503f-4d3f-87d4-6959703ee42d"). InnerVolumeSpecName "kube-api-access-hfzdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:12:16.764933 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.764916    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40bd54f-a2fe-409c-8396-c2707b94ac95-kube-api-access-cw75g" (OuterVolumeSpecName: "kube-api-access-cw75g") pod "f40bd54f-a2fe-409c-8396-c2707b94ac95" (UID: "f40bd54f-a2fe-409c-8396-c2707b94ac95"). InnerVolumeSpecName "kube-api-access-cw75g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:12:16.863040 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.862981    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5509762-503f-4d3f-87d4-6959703ee42d-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:16.863040 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.863011    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfzdl\" (UniqueName: \"kubernetes.io/projected/f5509762-503f-4d3f-87d4-6959703ee42d-kube-api-access-hfzdl\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:16.863040 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.863023    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9236229-2cc7-4272-89ff-70d0082dddc9-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:16.863040 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.863031    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bswd\" (UniqueName: \"kubernetes.io/projected/a9236229-2cc7-4272-89ff-70d0082dddc9-kube-api-access-9bswd\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:16.863040 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.863039    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40bd54f-a2fe-409c-8396-c2707b94ac95-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:16.863234 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.863048    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5509762-503f-4d3f-87d4-6959703ee42d-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:16.863234 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.863057    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw75g\" (UniqueName: \"kubernetes.io/projected/f40bd54f-a2fe-409c-8396-c2707b94ac95-kube-api-access-cw75g\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:16.895769 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:16.895738    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:17.146043 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.145928    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:17.368439 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.368407    2556 generic.go:356] "Generic (PLEG): container finished" podID="f40bd54f-a2fe-409c-8396-c2707b94ac95" containerID="d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b" exitCode=0
Jul 31 02:12:17.368855 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.368450    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"
Jul 31 02:12:17.368855 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.368501    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p" event={"ID":"f40bd54f-a2fe-409c-8396-c2707b94ac95","Type":"ContainerDied","Data":"d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b"}
Jul 31 02:12:17.368855 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.368548    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p" event={"ID":"f40bd54f-a2fe-409c-8396-c2707b94ac95","Type":"ContainerDied","Data":"491de0752faaafd99f11c0382a02598923d058cb1e4fe269c5f8676b0f583f7b"}
Jul 31 02:12:17.368855 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.368598    2556 scope.go:117] "RemoveContainer" containerID="d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b"
Jul 31 02:12:17.369968 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.369942    2556 generic.go:356] "Generic (PLEG): container finished" podID="a9236229-2cc7-4272-89ff-70d0082dddc9" containerID="f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94" exitCode=0
Jul 31 02:12:17.370088 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.369997    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"
Jul 31 02:12:17.370088 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.370035    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2" event={"ID":"a9236229-2cc7-4272-89ff-70d0082dddc9","Type":"ContainerDied","Data":"f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94"}
Jul 31 02:12:17.370366 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.370088    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2" event={"ID":"a9236229-2cc7-4272-89ff-70d0082dddc9","Type":"ContainerDied","Data":"de465c5e67e8d1ab873bba490fdf5fcae934f57c2ac17e0fc7647afd0110249d"}
Jul 31 02:12:17.371542 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.371518    2556 generic.go:356] "Generic (PLEG): container finished" podID="f5509762-503f-4d3f-87d4-6959703ee42d" containerID="8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6" exitCode=0
Jul 31 02:12:17.371673 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.371617    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"
Jul 31 02:12:17.371673 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.371659    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9" event={"ID":"f5509762-503f-4d3f-87d4-6959703ee42d","Type":"ContainerDied","Data":"8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6"}
Jul 31 02:12:17.371783 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.371699    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9" event={"ID":"f5509762-503f-4d3f-87d4-6959703ee42d","Type":"ContainerDied","Data":"b2bf4f37248e2dc3796cb24f93acc04483df0e23c00e54767b051c4eee7bd88a"}
Jul 31 02:12:17.378106 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.378050    2556 scope.go:117] "RemoveContainer" containerID="d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b"
Jul 31 02:12:17.378347 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:12:17.378326    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b\": container with ID starting with d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b not found: ID does not exist" containerID="d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b"
Jul 31 02:12:17.378428 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.378354    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b"} err="failed to get container status \"d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b\": rpc error: code = NotFound desc = could not find container \"d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b\": container with ID starting with d7603a1c49400da5d09603caf512d2d8a526faeed7f759c16f611b92a1aedf7b not found: ID does not exist"
Jul 31 02:12:17.378428 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.378370    2556 scope.go:117] "RemoveContainer" containerID="f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94"
Jul 31 02:12:17.386889 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.386872    2556 scope.go:117] "RemoveContainer" containerID="f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94"
Jul 31 02:12:17.387149 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:12:17.387129    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94\": container with ID starting with f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94 not found: ID does not exist" containerID="f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94"
Jul 31 02:12:17.387231 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.387157    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94"} err="failed to get container status \"f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94\": rpc error: code = NotFound desc = could not find container \"f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94\": container with ID starting with f1ded98be4c35be844b89634b773da4663b89d7a86d2def77249dbf967acaf94 not found: ID does not exist"
Jul 31 02:12:17.387231 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.387178    2556 scope.go:117] "RemoveContainer" containerID="8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6"
Jul 31 02:12:17.394625 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.394605    2556 scope.go:117] "RemoveContainer" containerID="8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6"
Jul 31 02:12:17.394874 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:12:17.394854    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6\": container with ID starting with 8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6 not found: ID does not exist" containerID="8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6"
Jul 31 02:12:17.394939 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.394878    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6"} err="failed to get container status \"8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6\": rpc error: code = NotFound desc = could not find container \"8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6\": container with ID starting with 8e8e0d9189a376ebace923a0c2103670471aa1824bb2147d4f8c56da10125ba6 not found: ID does not exist"
Jul 31 02:12:17.399097 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.399049    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"]
Jul 31 02:12:17.406960 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.406930    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-241ce02f-6b5dd4d4fb-6vp8p"]
Jul 31 02:12:17.424016 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.423963    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"]
Jul 31 02:12:17.435537 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.432955    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-241ce02f-5676948f69-6rrx9"]
Jul 31 02:12:17.444089 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.444064    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"]
Jul 31 02:12:17.451770 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.451749    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-241ce02f-6c9fcd9df7-6v8v2"]
Jul 31 02:12:17.452390 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.452354    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:17.757465 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:17.757384    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:18.151856 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:18.151827    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e5f98ce-6758-4b66-b9f1-ae0b1db76675" path="/var/lib/kubelet/pods/5e5f98ce-6758-4b66-b9f1-ae0b1db76675/volumes"
Jul 31 02:12:18.152167 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:18.152154    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9236229-2cc7-4272-89ff-70d0082dddc9" path="/var/lib/kubelet/pods/a9236229-2cc7-4272-89ff-70d0082dddc9/volumes"
Jul 31 02:12:18.152454 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:18.152443    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f40bd54f-a2fe-409c-8396-c2707b94ac95" path="/var/lib/kubelet/pods/f40bd54f-a2fe-409c-8396-c2707b94ac95/volumes"
Jul 31 02:12:18.152796 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:18.152784    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5509762-503f-4d3f-87d4-6959703ee42d" path="/var/lib/kubelet/pods/f5509762-503f-4d3f-87d4-6959703ee42d/volumes"
Jul 31 02:12:19.694828 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:19.694798    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:19.794721 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:19.794692    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:20.785385 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.785351    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"]
Jul 31 02:12:20.786702 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.786675    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e5f98ce-6758-4b66-b9f1-ae0b1db76675" containerName="revoke-keys"
Jul 31 02:12:20.786858 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.786844    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e5f98ce-6758-4b66-b9f1-ae0b1db76675" containerName="revoke-keys"
Jul 31 02:12:20.786992 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.786978    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5509762-503f-4d3f-87d4-6959703ee42d" containerName="maas-api"
Jul 31 02:12:20.787100 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.787088    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5509762-503f-4d3f-87d4-6959703ee42d" containerName="maas-api"
Jul 31 02:12:20.787199 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.787187    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40bd54f-a2fe-409c-8396-c2707b94ac95" containerName="payload-pre-processing"
Jul 31 02:12:20.787278 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.787268    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40bd54f-a2fe-409c-8396-c2707b94ac95" containerName="payload-pre-processing"
Jul 31 02:12:20.787395 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.787383    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9236229-2cc7-4272-89ff-70d0082dddc9" containerName="payload-processing"
Jul 31 02:12:20.787473 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.787463    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9236229-2cc7-4272-89ff-70d0082dddc9" containerName="payload-processing"
Jul 31 02:12:20.787737 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.787718    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9236229-2cc7-4272-89ff-70d0082dddc9" containerName="payload-processing"
Jul 31 02:12:20.787859 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.787846    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f40bd54f-a2fe-409c-8396-c2707b94ac95" containerName="payload-pre-processing"
Jul 31 02:12:20.787956 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.787944    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e5f98ce-6758-4b66-b9f1-ae0b1db76675" containerName="revoke-keys"
Jul 31 02:12:20.788044 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.788033    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5509762-503f-4d3f-87d4-6959703ee42d" containerName="maas-api"
Jul 31 02:12:20.795716 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.795697    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:20.798075 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.798055    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ebac695f-serving-cert\""
Jul 31 02:12:20.799512 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.799492    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"]
Jul 31 02:12:20.826474 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.826452    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:20.892715 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.892667    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ee2f97c-cf09-41de-902b-097d45ca9b54-maas-api-tls\") pod \"maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm\" (UID: \"2ee2f97c-cf09-41de-902b-097d45ca9b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:20.892873 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.892851    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595gv\" (UniqueName: \"kubernetes.io/projected/2ee2f97c-cf09-41de-902b-097d45ca9b54-kube-api-access-595gv\") pod \"maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm\" (UID: \"2ee2f97c-cf09-41de-902b-097d45ca9b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:20.893005 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.892938    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee2f97c-cf09-41de-902b-097d45ca9b54-openshift-service-ca\") pod \"maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm\" (UID: \"2ee2f97c-cf09-41de-902b-097d45ca9b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:20.938717 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.938691    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:20.994030 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.994006    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee2f97c-cf09-41de-902b-097d45ca9b54-openshift-service-ca\") pod \"maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm\" (UID: \"2ee2f97c-cf09-41de-902b-097d45ca9b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:20.994121 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.994062    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ee2f97c-cf09-41de-902b-097d45ca9b54-maas-api-tls\") pod \"maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm\" (UID: \"2ee2f97c-cf09-41de-902b-097d45ca9b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:20.994121 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.994084    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595gv\" (UniqueName: \"kubernetes.io/projected/2ee2f97c-cf09-41de-902b-097d45ca9b54-kube-api-access-595gv\") pod \"maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm\" (UID: \"2ee2f97c-cf09-41de-902b-097d45ca9b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:20.994610 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.994589    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee2f97c-cf09-41de-902b-097d45ca9b54-openshift-service-ca\") pod \"maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm\" (UID: \"2ee2f97c-cf09-41de-902b-097d45ca9b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:20.996434 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:20.996416    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ee2f97c-cf09-41de-902b-097d45ca9b54-maas-api-tls\") pod \"maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm\" (UID: \"2ee2f97c-cf09-41de-902b-097d45ca9b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:21.001831 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.001802    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-595gv\" (UniqueName: \"kubernetes.io/projected/2ee2f97c-cf09-41de-902b-097d45ca9b54-kube-api-access-595gv\") pod \"maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm\" (UID: \"2ee2f97c-cf09-41de-902b-097d45ca9b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:21.108300 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.108271    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:21.239730 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.239705    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"]
Jul 31 02:12:21.242363 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:12:21.242334    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee2f97c_cf09_41de_902b_097d45ca9b54.slice/crio-59fcbec1e706d6f5d159c7f82a9a68990dfb618eaf5f2acb1f4295a9ce3d539f WatchSource:0}: Error finding container 59fcbec1e706d6f5d159c7f82a9a68990dfb618eaf5f2acb1f4295a9ce3d539f: Status 404 returned error can't find the container with id 59fcbec1e706d6f5d159c7f82a9a68990dfb618eaf5f2acb1f4295a9ce3d539f
Jul 31 02:12:21.319359 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.319304    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:21.394828 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.394755    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm" event={"ID":"2ee2f97c-cf09-41de-902b-097d45ca9b54","Type":"ContainerStarted","Data":"59fcbec1e706d6f5d159c7f82a9a68990dfb618eaf5f2acb1f4295a9ce3d539f"}
Jul 31 02:12:21.407329 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.407300    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:21.412403 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.412382    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"]
Jul 31 02:12:21.416783 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.416767    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"
Jul 31 02:12:21.419209 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.419189    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ebac695f\""
Jul 31 02:12:21.419290 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.419198    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ebac695f-dockercfg-t6shj\""
Jul 31 02:12:21.424459 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.424329    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"]
Jul 31 02:12:21.446345 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.446322    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"]
Jul 31 02:12:21.449450 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.449431    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"
Jul 31 02:12:21.459498 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.459477    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"]
Jul 31 02:12:21.497955 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.497931    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jps9m\" (UniqueName: \"kubernetes.io/projected/a312726d-defd-40db-90a8-0d8f9bd7e168-kube-api-access-jps9m\") pod \"payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc\" (UID: \"a312726d-defd-40db-90a8-0d8f9bd7e168\") " pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"
Jul 31 02:12:21.498045 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.497976    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e92a687-759b-45fb-83c1-e15ce0acab99-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz\" (UID: \"7e92a687-759b-45fb-83c1-e15ce0acab99\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"
Jul 31 02:12:21.498045 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.498006    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkq49\" (UniqueName: \"kubernetes.io/projected/7e92a687-759b-45fb-83c1-e15ce0acab99-kube-api-access-dkq49\") pod \"payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz\" (UID: \"7e92a687-759b-45fb-83c1-e15ce0acab99\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"
Jul 31 02:12:21.498045 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.498042    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a312726d-defd-40db-90a8-0d8f9bd7e168-plugins-config-volume\") pod \"payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc\" (UID: \"a312726d-defd-40db-90a8-0d8f9bd7e168\") " pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"
Jul 31 02:12:21.598963 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.598937    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkq49\" (UniqueName: \"kubernetes.io/projected/7e92a687-759b-45fb-83c1-e15ce0acab99-kube-api-access-dkq49\") pod \"payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz\" (UID: \"7e92a687-759b-45fb-83c1-e15ce0acab99\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"
Jul 31 02:12:21.599113 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.598972    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a312726d-defd-40db-90a8-0d8f9bd7e168-plugins-config-volume\") pod \"payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc\" (UID: \"a312726d-defd-40db-90a8-0d8f9bd7e168\") " pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"
Jul 31 02:12:21.599113 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.599047    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jps9m\" (UniqueName: \"kubernetes.io/projected/a312726d-defd-40db-90a8-0d8f9bd7e168-kube-api-access-jps9m\") pod \"payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc\" (UID: \"a312726d-defd-40db-90a8-0d8f9bd7e168\") " pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"
Jul 31 02:12:21.599113 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.599107    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e92a687-759b-45fb-83c1-e15ce0acab99-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz\" (UID: \"7e92a687-759b-45fb-83c1-e15ce0acab99\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"
Jul 31 02:12:21.599790 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.599772    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e92a687-759b-45fb-83c1-e15ce0acab99-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz\" (UID: \"7e92a687-759b-45fb-83c1-e15ce0acab99\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"
Jul 31 02:12:21.599855 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.599842    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a312726d-defd-40db-90a8-0d8f9bd7e168-plugins-config-volume\") pod \"payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc\" (UID: \"a312726d-defd-40db-90a8-0d8f9bd7e168\") " pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"
Jul 31 02:12:21.606423 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.606402    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkq49\" (UniqueName: \"kubernetes.io/projected/7e92a687-759b-45fb-83c1-e15ce0acab99-kube-api-access-dkq49\") pod \"payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz\" (UID: \"7e92a687-759b-45fb-83c1-e15ce0acab99\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"
Jul 31 02:12:21.606509 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.606450    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jps9m\" (UniqueName: \"kubernetes.io/projected/a312726d-defd-40db-90a8-0d8f9bd7e168-kube-api-access-jps9m\") pod \"payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc\" (UID: \"a312726d-defd-40db-90a8-0d8f9bd7e168\") " pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"
Jul 31 02:12:21.727740 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.727720    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"
Jul 31 02:12:21.759597 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.759546    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"
Jul 31 02:12:21.866064 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.866032    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"]
Jul 31 02:12:21.869833 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:12:21.869798    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e92a687_759b_45fb_83c1_e15ce0acab99.slice/crio-c5a9a1bf032a334c6dc85cdc8c3e4b4e01c2bd04c7da760d745794c316a91d4a WatchSource:0}: Error finding container c5a9a1bf032a334c6dc85cdc8c3e4b4e01c2bd04c7da760d745794c316a91d4a: Status 404 returned error can't find the container with id c5a9a1bf032a334c6dc85cdc8c3e4b4e01c2bd04c7da760d745794c316a91d4a
Jul 31 02:12:21.905109 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:21.904794    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"]
Jul 31 02:12:21.907151 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:12:21.907126    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda312726d_defd_40db_90a8_0d8f9bd7e168.slice/crio-d3ccedd6ac9fa89a309b9d27b81c44f6914e6e2571c2ffc77c86224e68d3c2c3 WatchSource:0}: Error finding container d3ccedd6ac9fa89a309b9d27b81c44f6914e6e2571c2ffc77c86224e68d3c2c3: Status 404 returned error can't find the container with id d3ccedd6ac9fa89a309b9d27b81c44f6914e6e2571c2ffc77c86224e68d3c2c3
Jul 31 02:12:22.401067 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:22.401027    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz" event={"ID":"7e92a687-759b-45fb-83c1-e15ce0acab99","Type":"ContainerStarted","Data":"c0e212d886dedbb24e1e1f2d4789853d5bd25ca9c095a36e4b10878c4543c021"}
Jul 31 02:12:22.401284 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:22.401106    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz" event={"ID":"7e92a687-759b-45fb-83c1-e15ce0acab99","Type":"ContainerStarted","Data":"c5a9a1bf032a334c6dc85cdc8c3e4b4e01c2bd04c7da760d745794c316a91d4a"}
Jul 31 02:12:22.401284 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:22.401140    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"
Jul 31 02:12:22.402811 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:22.402781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm" event={"ID":"2ee2f97c-cf09-41de-902b-097d45ca9b54","Type":"ContainerStarted","Data":"1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b"}
Jul 31 02:12:22.402945 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:22.402840    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:22.404366 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:22.404344    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc" event={"ID":"a312726d-defd-40db-90a8-0d8f9bd7e168","Type":"ContainerStarted","Data":"614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6"}
Jul 31 02:12:22.404459 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:22.404381    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc" event={"ID":"a312726d-defd-40db-90a8-0d8f9bd7e168","Type":"ContainerStarted","Data":"d3ccedd6ac9fa89a309b9d27b81c44f6914e6e2571c2ffc77c86224e68d3c2c3"}
Jul 31 02:12:22.404459 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:22.404412    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"
Jul 31 02:12:22.418662 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:22.418538    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz" podStartSLOduration=1.41852117 podStartE2EDuration="1.41852117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:12:22.4167576 +0000 UTC m=+2100.782503229" watchObservedRunningTime="2026-07-31 02:12:22.41852117 +0000 UTC m=+2100.784266801"
Jul 31 02:12:22.436718 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:22.436677    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm" podStartSLOduration=2.026937534 podStartE2EDuration="2.436664918s" totalImagesPullingTime="409.727384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:12:22.43583179 +0000 UTC m=+2100.801577455" watchObservedRunningTime="2026-07-31 02:12:22.436664918 +0000 UTC m=+2100.802410547"
Jul 31 02:12:22.450492 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:22.450456    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc" podStartSLOduration=1.450444038 podStartE2EDuration="1.450444038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:12:22.449512709 +0000 UTC m=+2100.815258337" watchObservedRunningTime="2026-07-31 02:12:22.450444038 +0000 UTC m=+2100.816189707"
Jul 31 02:12:28.414738 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:28.414711    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:33.411933 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:33.411900    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"
Jul 31 02:12:33.412428 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:33.411976    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"
Jul 31 02:12:45.576160 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:45.576122    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298"]
Jul 31 02:12:45.579735 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:45.579712    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298"
Jul 31 02:12:45.581780 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:45.581761    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:12:45.585665 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:45.585634    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298"]
Jul 31 02:12:45.691907 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:45.691861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1785a3-0b8c-4c5c-bbed-cd8b93e02588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebac695f-fb298\" (UID: \"bf1785a3-0b8c-4c5c-bbed-cd8b93e02588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298"
Jul 31 02:12:45.793214 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:45.793171    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1785a3-0b8c-4c5c-bbed-cd8b93e02588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebac695f-fb298\" (UID: \"bf1785a3-0b8c-4c5c-bbed-cd8b93e02588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298"
Jul 31 02:12:45.793828 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:45.793810    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1785a3-0b8c-4c5c-bbed-cd8b93e02588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebac695f-fb298\" (UID: \"bf1785a3-0b8c-4c5c-bbed-cd8b93e02588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298"
Jul 31 02:12:45.892181 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:45.892106    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298"
Jul 31 02:12:46.021078 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:46.021046    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298"]
Jul 31 02:12:46.024940 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:12:46.024914    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1785a3_0b8c_4c5c_bbed_cd8b93e02588.slice/crio-3a63cdf26bdad61fc9a038b67898cc73f0332f36826fa7b14a377a84cae0c081 WatchSource:0}: Error finding container 3a63cdf26bdad61fc9a038b67898cc73f0332f36826fa7b14a377a84cae0c081: Status 404 returned error can't find the container with id 3a63cdf26bdad61fc9a038b67898cc73f0332f36826fa7b14a377a84cae0c081
Jul 31 02:12:46.516347 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:46.516233    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298" event={"ID":"bf1785a3-0b8c-4c5c-bbed-cd8b93e02588","Type":"ContainerStarted","Data":"167ed80f138833bfe76012d866a066b899f68e79d7c38122716e72c06b3d6e61"}
Jul 31 02:12:46.516513 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:46.516353    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298" event={"ID":"bf1785a3-0b8c-4c5c-bbed-cd8b93e02588","Type":"ContainerStarted","Data":"3a63cdf26bdad61fc9a038b67898cc73f0332f36826fa7b14a377a84cae0c081"}
Jul 31 02:12:46.529342 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:46.529285    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298" podStartSLOduration=1.529270102 podStartE2EDuration="1.529270102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:12:46.528763217 +0000 UTC m=+2124.894508859" watchObservedRunningTime="2026-07-31 02:12:46.529270102 +0000 UTC m=+2124.895015732"
Jul 31 02:12:47.523057 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:47.523017    2556 generic.go:356] "Generic (PLEG): container finished" podID="bf1785a3-0b8c-4c5c-bbed-cd8b93e02588" containerID="167ed80f138833bfe76012d866a066b899f68e79d7c38122716e72c06b3d6e61" exitCode=0
Jul 31 02:12:47.523456 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:47.523099    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298" event={"ID":"bf1785a3-0b8c-4c5c-bbed-cd8b93e02588","Type":"ContainerDied","Data":"167ed80f138833bfe76012d866a066b899f68e79d7c38122716e72c06b3d6e61"}
Jul 31 02:12:48.659542 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:48.659515    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298"
Jul 31 02:12:48.820167 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:48.820141    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1785a3-0b8c-4c5c-bbed-cd8b93e02588-maas-api-service-ca\") pod \"bf1785a3-0b8c-4c5c-bbed-cd8b93e02588\" (UID: \"bf1785a3-0b8c-4c5c-bbed-cd8b93e02588\") "
Jul 31 02:12:48.820494 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:48.820467    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1785a3-0b8c-4c5c-bbed-cd8b93e02588-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bf1785a3-0b8c-4c5c-bbed-cd8b93e02588" (UID: "bf1785a3-0b8c-4c5c-bbed-cd8b93e02588"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:12:48.921345 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:48.921324    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1785a3-0b8c-4c5c-bbed-cd8b93e02588-maas-api-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:49.533634 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:49.533600    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298" event={"ID":"bf1785a3-0b8c-4c5c-bbed-cd8b93e02588","Type":"ContainerDied","Data":"3a63cdf26bdad61fc9a038b67898cc73f0332f36826fa7b14a377a84cae0c081"}
Jul 31 02:12:49.533806 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:49.533671    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a63cdf26bdad61fc9a038b67898cc73f0332f36826fa7b14a377a84cae0c081"
Jul 31 02:12:49.533806 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:49.533612    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298"
Jul 31 02:12:51.209163 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.209124    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"]
Jul 31 02:12:51.209634 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.209375    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm" podUID="2ee2f97c-cf09-41de-902b-097d45ca9b54" containerName="maas-api" containerID="cri-o://1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b" gracePeriod=30
Jul 31 02:12:51.236382 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.236346    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:51.263574 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.263533    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"]
Jul 31 02:12:51.263841 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.263784    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc" podUID="a312726d-defd-40db-90a8-0d8f9bd7e168" containerName="payload-processing" containerID="cri-o://614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6" gracePeriod=30
Jul 31 02:12:51.268257 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.268216    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"]
Jul 31 02:12:51.268971 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.268906    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz" podUID="7e92a687-759b-45fb-83c1-e15ce0acab99" containerName="payload-pre-processing" containerID="cri-o://c0e212d886dedbb24e1e1f2d4789853d5bd25ca9c095a36e4b10878c4543c021" gracePeriod=30
Jul 31 02:12:51.308908 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.308877    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:51.363128 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.363100    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:51.401527 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.401494    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298"]
Jul 31 02:12:51.405362 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.405231    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebac695f-fb298"]
Jul 31 02:12:51.419271 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.419227    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:51.513860 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.513836    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:51.542705 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.542683    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"
Jul 31 02:12:51.544585 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.544544    2556 generic.go:356] "Generic (PLEG): container finished" podID="2ee2f97c-cf09-41de-902b-097d45ca9b54" containerID="1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b" exitCode=0
Jul 31 02:12:51.544707 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.544594    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm" event={"ID":"2ee2f97c-cf09-41de-902b-097d45ca9b54","Type":"ContainerDied","Data":"1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b"}
Jul 31 02:12:51.544707 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.544626    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"
Jul 31 02:12:51.544707 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.544666    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm" event={"ID":"2ee2f97c-cf09-41de-902b-097d45ca9b54","Type":"ContainerDied","Data":"59fcbec1e706d6f5d159c7f82a9a68990dfb618eaf5f2acb1f4295a9ce3d539f"}
Jul 31 02:12:51.544707 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.544699    2556 scope.go:117] "RemoveContainer" containerID="1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b"
Jul 31 02:12:51.546237 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.546217    2556 generic.go:356] "Generic (PLEG): container finished" podID="a312726d-defd-40db-90a8-0d8f9bd7e168" containerID="614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6" exitCode=0
Jul 31 02:12:51.546396 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.546330    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"
Jul 31 02:12:51.546396 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.546355    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc" event={"ID":"a312726d-defd-40db-90a8-0d8f9bd7e168","Type":"ContainerDied","Data":"614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6"}
Jul 31 02:12:51.546591 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.546409    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc" event={"ID":"a312726d-defd-40db-90a8-0d8f9bd7e168","Type":"ContainerDied","Data":"d3ccedd6ac9fa89a309b9d27b81c44f6914e6e2571c2ffc77c86224e68d3c2c3"}
Jul 31 02:12:51.548163 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.548134    2556 generic.go:356] "Generic (PLEG): container finished" podID="7e92a687-759b-45fb-83c1-e15ce0acab99" containerID="c0e212d886dedbb24e1e1f2d4789853d5bd25ca9c095a36e4b10878c4543c021" exitCode=0
Jul 31 02:12:51.548163 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.548163    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz" event={"ID":"7e92a687-759b-45fb-83c1-e15ce0acab99","Type":"ContainerDied","Data":"c0e212d886dedbb24e1e1f2d4789853d5bd25ca9c095a36e4b10878c4543c021"}
Jul 31 02:12:51.554093 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.554077    2556 scope.go:117] "RemoveContainer" containerID="1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b"
Jul 31 02:12:51.554370 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:12:51.554352    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b\": container with ID starting with 1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b not found: ID does not exist" containerID="1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b"
Jul 31 02:12:51.554463 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.554383    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b"} err="failed to get container status \"1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b\": rpc error: code = NotFound desc = could not find container \"1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b\": container with ID starting with 1768e516f13f8ee5ed694ede710b4a58fd2942487c42b377b1e8148c05c53c2b not found: ID does not exist"
Jul 31 02:12:51.554463 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.554406    2556 scope.go:117] "RemoveContainer" containerID="614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6"
Jul 31 02:12:51.564369 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.564344    2556 scope.go:117] "RemoveContainer" containerID="614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6"
Jul 31 02:12:51.564651 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:12:51.564618    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6\": container with ID starting with 614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6 not found: ID does not exist" containerID="614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6"
Jul 31 02:12:51.564751 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.564664    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6"} err="failed to get container status \"614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6\": rpc error: code = NotFound desc = could not find container \"614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6\": container with ID starting with 614f80041e190b0c5631d8b6019e55d76ead58360ba31aa187f2e8de5002a6a6 not found: ID does not exist"
Jul 31 02:12:51.568729 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.568711    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"
Jul 31 02:12:51.646259 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.646232    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-595gv\" (UniqueName: \"kubernetes.io/projected/2ee2f97c-cf09-41de-902b-097d45ca9b54-kube-api-access-595gv\") pod \"2ee2f97c-cf09-41de-902b-097d45ca9b54\" (UID: \"2ee2f97c-cf09-41de-902b-097d45ca9b54\") "
Jul 31 02:12:51.646402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.646289    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ee2f97c-cf09-41de-902b-097d45ca9b54-maas-api-tls\") pod \"2ee2f97c-cf09-41de-902b-097d45ca9b54\" (UID: \"2ee2f97c-cf09-41de-902b-097d45ca9b54\") "
Jul 31 02:12:51.646402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.646324    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee2f97c-cf09-41de-902b-097d45ca9b54-openshift-service-ca\") pod \"2ee2f97c-cf09-41de-902b-097d45ca9b54\" (UID: \"2ee2f97c-cf09-41de-902b-097d45ca9b54\") "
Jul 31 02:12:51.646402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.646352    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jps9m\" (UniqueName: \"kubernetes.io/projected/a312726d-defd-40db-90a8-0d8f9bd7e168-kube-api-access-jps9m\") pod \"a312726d-defd-40db-90a8-0d8f9bd7e168\" (UID: \"a312726d-defd-40db-90a8-0d8f9bd7e168\") "
Jul 31 02:12:51.646402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.646376    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a312726d-defd-40db-90a8-0d8f9bd7e168-plugins-config-volume\") pod \"a312726d-defd-40db-90a8-0d8f9bd7e168\" (UID: \"a312726d-defd-40db-90a8-0d8f9bd7e168\") "
Jul 31 02:12:51.646759 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.646730    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee2f97c-cf09-41de-902b-097d45ca9b54-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ee2f97c-cf09-41de-902b-097d45ca9b54" (UID: "2ee2f97c-cf09-41de-902b-097d45ca9b54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:12:51.646838 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.646809    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a312726d-defd-40db-90a8-0d8f9bd7e168-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a312726d-defd-40db-90a8-0d8f9bd7e168" (UID: "a312726d-defd-40db-90a8-0d8f9bd7e168"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:12:51.648515 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.648489    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ee2f97c-cf09-41de-902b-097d45ca9b54-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ee2f97c-cf09-41de-902b-097d45ca9b54" (UID: "2ee2f97c-cf09-41de-902b-097d45ca9b54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:12:51.648634 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.648543    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a312726d-defd-40db-90a8-0d8f9bd7e168-kube-api-access-jps9m" (OuterVolumeSpecName: "kube-api-access-jps9m") pod "a312726d-defd-40db-90a8-0d8f9bd7e168" (UID: "a312726d-defd-40db-90a8-0d8f9bd7e168"). InnerVolumeSpecName "kube-api-access-jps9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:12:51.648634 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.648537    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee2f97c-cf09-41de-902b-097d45ca9b54-kube-api-access-595gv" (OuterVolumeSpecName: "kube-api-access-595gv") pod "2ee2f97c-cf09-41de-902b-097d45ca9b54" (UID: "2ee2f97c-cf09-41de-902b-097d45ca9b54"). InnerVolumeSpecName "kube-api-access-595gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:12:51.747504 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.747429    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e92a687-759b-45fb-83c1-e15ce0acab99-plugins-config-volume\") pod \"7e92a687-759b-45fb-83c1-e15ce0acab99\" (UID: \"7e92a687-759b-45fb-83c1-e15ce0acab99\") "
Jul 31 02:12:51.747504 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.747470    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkq49\" (UniqueName: \"kubernetes.io/projected/7e92a687-759b-45fb-83c1-e15ce0acab99-kube-api-access-dkq49\") pod \"7e92a687-759b-45fb-83c1-e15ce0acab99\" (UID: \"7e92a687-759b-45fb-83c1-e15ce0acab99\") "
Jul 31 02:12:51.747854 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.747822    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e92a687-759b-45fb-83c1-e15ce0acab99-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e92a687-759b-45fb-83c1-e15ce0acab99" (UID: "7e92a687-759b-45fb-83c1-e15ce0acab99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:12:51.747854 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.747829    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-595gv\" (UniqueName: \"kubernetes.io/projected/2ee2f97c-cf09-41de-902b-097d45ca9b54-kube-api-access-595gv\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:51.747986 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.747867    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ee2f97c-cf09-41de-902b-097d45ca9b54-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:51.747986 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.747879    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee2f97c-cf09-41de-902b-097d45ca9b54-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:51.747986 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.747889    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jps9m\" (UniqueName: \"kubernetes.io/projected/a312726d-defd-40db-90a8-0d8f9bd7e168-kube-api-access-jps9m\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:51.747986 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.747897    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a312726d-defd-40db-90a8-0d8f9bd7e168-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:51.749420 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.749405    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e92a687-759b-45fb-83c1-e15ce0acab99-kube-api-access-dkq49" (OuterVolumeSpecName: "kube-api-access-dkq49") pod "7e92a687-759b-45fb-83c1-e15ce0acab99" (UID: "7e92a687-759b-45fb-83c1-e15ce0acab99"). InnerVolumeSpecName "kube-api-access-dkq49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:12:51.849265 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.849235    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e92a687-759b-45fb-83c1-e15ce0acab99-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:51.849355 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.849266    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkq49\" (UniqueName: \"kubernetes.io/projected/7e92a687-759b-45fb-83c1-e15ce0acab99-kube-api-access-dkq49\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:12:51.866984 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.866953    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"]
Jul 31 02:12:51.870317 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.870291    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebac695f-cc5759c4b-rbwdm"]
Jul 31 02:12:51.879747 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.879723    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"]
Jul 31 02:12:51.882832 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:51.882813    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebac695f-6677b8cfbb-mxkrc"]
Jul 31 02:12:52.151810 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:52.151782    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ee2f97c-cf09-41de-902b-097d45ca9b54" path="/var/lib/kubelet/pods/2ee2f97c-cf09-41de-902b-097d45ca9b54/volumes"
Jul 31 02:12:52.152106 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:52.152092    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a312726d-defd-40db-90a8-0d8f9bd7e168" path="/var/lib/kubelet/pods/a312726d-defd-40db-90a8-0d8f9bd7e168/volumes"
Jul 31 02:12:52.152365 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:52.152354    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf1785a3-0b8c-4c5c-bbed-cd8b93e02588" path="/var/lib/kubelet/pods/bf1785a3-0b8c-4c5c-bbed-cd8b93e02588/volumes"
Jul 31 02:12:52.553965 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:52.553886    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"
Jul 31 02:12:52.554400 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:52.553887    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz" event={"ID":"7e92a687-759b-45fb-83c1-e15ce0acab99","Type":"ContainerDied","Data":"c5a9a1bf032a334c6dc85cdc8c3e4b4e01c2bd04c7da760d745794c316a91d4a"}
Jul 31 02:12:52.554400 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:52.554074    2556 scope.go:117] "RemoveContainer" containerID="c0e212d886dedbb24e1e1f2d4789853d5bd25ca9c095a36e4b10878c4543c021"
Jul 31 02:12:52.571956 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:52.571884    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"]
Jul 31 02:12:52.577161 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:52.577133    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ebac695f-6cd48576d8-shfsz"]
Jul 31 02:12:52.604285 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:52.604263    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:52.827997 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:52.827959    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:53.056720 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:53.056688    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:53.378873 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:53.378840    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:53.534994 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:53.534956    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:54.151219 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:54.151184    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e92a687-759b-45fb-83c1-e15ce0acab99" path="/var/lib/kubelet/pods/7e92a687-759b-45fb-83c1-e15ce0acab99/volumes"
Jul 31 02:12:55.925430 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:55.925399    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:56.033619 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:56.033588    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:57.012004 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.011970    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"]
Jul 31 02:12:57.012350 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.012269    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee2f97c-cf09-41de-902b-097d45ca9b54" containerName="maas-api"
Jul 31 02:12:57.012350 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.012280    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee2f97c-cf09-41de-902b-097d45ca9b54" containerName="maas-api"
Jul 31 02:12:57.012350 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.012292    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e92a687-759b-45fb-83c1-e15ce0acab99" containerName="payload-pre-processing"
Jul 31 02:12:57.012350 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.012300    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e92a687-759b-45fb-83c1-e15ce0acab99" containerName="payload-pre-processing"
Jul 31 02:12:57.012350 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.012311    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1785a3-0b8c-4c5c-bbed-cd8b93e02588" containerName="revoke-keys"
Jul 31 02:12:57.012350 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.012317    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1785a3-0b8c-4c5c-bbed-cd8b93e02588" containerName="revoke-keys"
Jul 31 02:12:57.012350 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.012326    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a312726d-defd-40db-90a8-0d8f9bd7e168" containerName="payload-processing"
Jul 31 02:12:57.012350 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.012331    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a312726d-defd-40db-90a8-0d8f9bd7e168" containerName="payload-processing"
Jul 31 02:12:57.012636 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.012379    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e92a687-759b-45fb-83c1-e15ce0acab99" containerName="payload-pre-processing"
Jul 31 02:12:57.012636 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.012386    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf1785a3-0b8c-4c5c-bbed-cd8b93e02588" containerName="revoke-keys"
Jul 31 02:12:57.012636 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.012393    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee2f97c-cf09-41de-902b-097d45ca9b54" containerName="maas-api"
Jul 31 02:12:57.012636 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.012399    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a312726d-defd-40db-90a8-0d8f9bd7e168" containerName="payload-processing"
Jul 31 02:12:57.015395 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.015374    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:57.017381 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.017353    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-cb65761c-serving-cert\""
Jul 31 02:12:57.025776 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.025752    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"]
Jul 31 02:12:57.085770 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.085746    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:57.088648 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.088627    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5f5k\" (UniqueName: \"kubernetes.io/projected/ba3379a7-1432-4ec0-b4ac-82abacb8dede-kube-api-access-j5f5k\") pod \"maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:57.088768 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.088659    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3379a7-1432-4ec0-b4ac-82abacb8dede-openshift-service-ca\") pod \"maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:57.088768 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.088713    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba3379a7-1432-4ec0-b4ac-82abacb8dede-maas-api-tls\") pod \"maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:57.189611 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.189590    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5f5k\" (UniqueName: \"kubernetes.io/projected/ba3379a7-1432-4ec0-b4ac-82abacb8dede-kube-api-access-j5f5k\") pod \"maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:57.189710 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.189621    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3379a7-1432-4ec0-b4ac-82abacb8dede-openshift-service-ca\") pod \"maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:57.189710 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.189658    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba3379a7-1432-4ec0-b4ac-82abacb8dede-maas-api-tls\") pod \"maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:57.189843 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:12:57.189763    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-serving-cert: secret "maas-api-e2e-derive-cb65761c-serving-cert" not found
Jul 31 02:12:57.189843 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:12:57.189813    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba3379a7-1432-4ec0-b4ac-82abacb8dede-maas-api-tls podName:ba3379a7-1432-4ec0-b4ac-82abacb8dede nodeName:}" failed. No retries permitted until 2026-07-31 02:12:57.689795902 +0000 UTC m=+2136.055541513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba3379a7-1432-4ec0-b4ac-82abacb8dede-maas-api-tls") pod "maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x" (UID: "ba3379a7-1432-4ec0-b4ac-82abacb8dede") : secret "maas-api-e2e-derive-cb65761c-serving-cert" not found
Jul 31 02:12:57.190270 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.190248    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3379a7-1432-4ec0-b4ac-82abacb8dede-openshift-service-ca\") pod \"maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:57.199629 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.199603    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5f5k\" (UniqueName: \"kubernetes.io/projected/ba3379a7-1432-4ec0-b4ac-82abacb8dede-kube-api-access-j5f5k\") pod \"maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:57.423221 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.423201    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:57.530026 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.530002    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:57.640677 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.640645    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:12:57.641008 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.640992    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"]
Jul 31 02:12:57.644407 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.644389    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"
Jul 31 02:12:57.646575 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.646537    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-cb65761c-dockercfg-z92fc\""
Jul 31 02:12:57.646666 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.646607    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-cb65761c\""
Jul 31 02:12:57.653406 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.653384    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"]
Jul 31 02:12:57.656873 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.656853    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"]
Jul 31 02:12:57.656965 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.656953    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"
Jul 31 02:12:57.667788 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.667761    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"]
Jul 31 02:12:57.693038 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.692983    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18492aad-17c8-4a21-8497-b565aba7a111-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c\" (UID: \"18492aad-17c8-4a21-8497-b565aba7a111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"
Jul 31 02:12:57.693038 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.693015    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9tkc\" (UniqueName: \"kubernetes.io/projected/18492aad-17c8-4a21-8497-b565aba7a111-kube-api-access-w9tkc\") pod \"payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c\" (UID: \"18492aad-17c8-4a21-8497-b565aba7a111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"
Jul 31 02:12:57.693156 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.693055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e5552b-a118-425e-a432-4433d99e4332-plugins-config-volume\") pod \"payload-processing-e2e-derive-cb65761c-88646458b-928tw\" (UID: \"a0e5552b-a118-425e-a432-4433d99e4332\") " pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"
Jul 31 02:12:57.693156 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.693088    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4f8\" (UniqueName: \"kubernetes.io/projected/a0e5552b-a118-425e-a432-4433d99e4332-kube-api-access-8b4f8\") pod \"payload-processing-e2e-derive-cb65761c-88646458b-928tw\" (UID: \"a0e5552b-a118-425e-a432-4433d99e4332\") " pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"
Jul 31 02:12:57.693227 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.693163    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba3379a7-1432-4ec0-b4ac-82abacb8dede-maas-api-tls\") pod \"maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:57.695663 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.695644    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba3379a7-1432-4ec0-b4ac-82abacb8dede-maas-api-tls\") pod \"maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:57.794065 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.794038    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18492aad-17c8-4a21-8497-b565aba7a111-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c\" (UID: \"18492aad-17c8-4a21-8497-b565aba7a111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"
Jul 31 02:12:57.794189 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.794068    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9tkc\" (UniqueName: \"kubernetes.io/projected/18492aad-17c8-4a21-8497-b565aba7a111-kube-api-access-w9tkc\") pod \"payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c\" (UID: \"18492aad-17c8-4a21-8497-b565aba7a111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"
Jul 31 02:12:57.794189 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.794099    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e5552b-a118-425e-a432-4433d99e4332-plugins-config-volume\") pod \"payload-processing-e2e-derive-cb65761c-88646458b-928tw\" (UID: \"a0e5552b-a118-425e-a432-4433d99e4332\") " pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"
Jul 31 02:12:57.794189 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.794129    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4f8\" (UniqueName: \"kubernetes.io/projected/a0e5552b-a118-425e-a432-4433d99e4332-kube-api-access-8b4f8\") pod \"payload-processing-e2e-derive-cb65761c-88646458b-928tw\" (UID: \"a0e5552b-a118-425e-a432-4433d99e4332\") " pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"
Jul 31 02:12:57.794802 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.794785    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18492aad-17c8-4a21-8497-b565aba7a111-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c\" (UID: \"18492aad-17c8-4a21-8497-b565aba7a111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"
Jul 31 02:12:57.794976 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.794954    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e5552b-a118-425e-a432-4433d99e4332-plugins-config-volume\") pod \"payload-processing-e2e-derive-cb65761c-88646458b-928tw\" (UID: \"a0e5552b-a118-425e-a432-4433d99e4332\") " pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"
Jul 31 02:12:57.801986 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.801956    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4f8\" (UniqueName: \"kubernetes.io/projected/a0e5552b-a118-425e-a432-4433d99e4332-kube-api-access-8b4f8\") pod \"payload-processing-e2e-derive-cb65761c-88646458b-928tw\" (UID: \"a0e5552b-a118-425e-a432-4433d99e4332\") " pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"
Jul 31 02:12:57.802190 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.802171    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9tkc\" (UniqueName: \"kubernetes.io/projected/18492aad-17c8-4a21-8497-b565aba7a111-kube-api-access-w9tkc\") pod \"payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c\" (UID: \"18492aad-17c8-4a21-8497-b565aba7a111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"
Jul 31 02:12:57.927313 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.927283    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:57.955278 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.955223    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"
Jul 31 02:12:57.967708 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:57.967500    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"
Jul 31 02:12:58.087423 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:58.087392    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"]
Jul 31 02:12:58.092548 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:12:58.091883    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3379a7_1432_4ec0_b4ac_82abacb8dede.slice/crio-96229252cffe3ccc92b26d1c88f0eb37595cdb9cf5a98a29da7aab00233f87a5 WatchSource:0}: Error finding container 96229252cffe3ccc92b26d1c88f0eb37595cdb9cf5a98a29da7aab00233f87a5: Status 404 returned error can't find the container with id 96229252cffe3ccc92b26d1c88f0eb37595cdb9cf5a98a29da7aab00233f87a5
Jul 31 02:12:58.117332 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:58.117304    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"]
Jul 31 02:12:58.120421 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:12:58.120393    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18492aad_17c8_4a21_8497_b565aba7a111.slice/crio-96bf0dcb20c407d7570ebb0c461ed11e434d29968359198f3e2570c406125233 WatchSource:0}: Error finding container 96bf0dcb20c407d7570ebb0c461ed11e434d29968359198f3e2570c406125233: Status 404 returned error can't find the container with id 96bf0dcb20c407d7570ebb0c461ed11e434d29968359198f3e2570c406125233
Jul 31 02:12:58.142142 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:58.142097    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"]
Jul 31 02:12:58.150952 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:12:58.150886    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e5552b_a118_425e_a432_4433d99e4332.slice/crio-c8a8f4cc21bd19634e54abc02fade0521849e930bf17a2e3bc2b257a6c128e67 WatchSource:0}: Error finding container c8a8f4cc21bd19634e54abc02fade0521849e930bf17a2e3bc2b257a6c128e67: Status 404 returned error can't find the container with id c8a8f4cc21bd19634e54abc02fade0521849e930bf17a2e3bc2b257a6c128e67
Jul 31 02:12:58.589749 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:58.589716    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw" event={"ID":"a0e5552b-a118-425e-a432-4433d99e4332","Type":"ContainerStarted","Data":"0506fd23e560aab671492ce2dd80f3e3b112e878fb4a777c4cb175e806bdecaa"}
Jul 31 02:12:58.589927 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:58.589775    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw" event={"ID":"a0e5552b-a118-425e-a432-4433d99e4332","Type":"ContainerStarted","Data":"c8a8f4cc21bd19634e54abc02fade0521849e930bf17a2e3bc2b257a6c128e67"}
Jul 31 02:12:58.589927 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:58.589847    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"
Jul 31 02:12:58.591086 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:58.591062    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x" event={"ID":"ba3379a7-1432-4ec0-b4ac-82abacb8dede","Type":"ContainerStarted","Data":"96229252cffe3ccc92b26d1c88f0eb37595cdb9cf5a98a29da7aab00233f87a5"}
Jul 31 02:12:58.592488 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:58.592469    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c" event={"ID":"18492aad-17c8-4a21-8497-b565aba7a111","Type":"ContainerStarted","Data":"8d0b9358b0e90fec2776323b7c6d1b72a52866ce14f19372e9176ac92108a6b3"}
Jul 31 02:12:58.592611 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:58.592509    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c" event={"ID":"18492aad-17c8-4a21-8497-b565aba7a111","Type":"ContainerStarted","Data":"96bf0dcb20c407d7570ebb0c461ed11e434d29968359198f3e2570c406125233"}
Jul 31 02:12:58.592611 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:58.592527    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"
Jul 31 02:12:58.606146 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:58.606106    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw" podStartSLOduration=1.6060952130000001 podStartE2EDuration="1.606095213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:12:58.605592631 +0000 UTC m=+2136.971338256" watchObservedRunningTime="2026-07-31 02:12:58.606095213 +0000 UTC m=+2136.971840838"
Jul 31 02:12:58.622007 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:58.621969    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c" podStartSLOduration=1.621957964 podStartE2EDuration="1.621957964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:12:58.620092551 +0000 UTC m=+2136.985838180" watchObservedRunningTime="2026-07-31 02:12:58.621957964 +0000 UTC m=+2136.987703594"
Jul 31 02:12:59.598891 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:59.598850    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x" event={"ID":"ba3379a7-1432-4ec0-b4ac-82abacb8dede","Type":"ContainerStarted","Data":"a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8"}
Jul 31 02:12:59.599334 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:59.599193    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:12:59.614265 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:12:59.614215    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x" podStartSLOduration=3.028229824 podStartE2EDuration="3.614199274s" totalImagesPullingTime="585.96945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:12:59.613411312 +0000 UTC m=+2137.979156941" watchObservedRunningTime="2026-07-31 02:12:59.614199274 +0000 UTC m=+2137.979944912"
Jul 31 02:13:02.526042 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:02.526004    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:02.578455 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:02.578417    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:06.614236 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:06.614206    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:13:09.600500 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:09.600459    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c" podUID="18492aad-17c8-4a21-8497-b565aba7a111" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.41:9004: connect: connection refused"
Jul 31 02:13:09.600969 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:09.600930    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"
Jul 31 02:13:19.600375 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:19.600342    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"
Jul 31 02:13:25.965855 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:25.965820    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t"]
Jul 31 02:13:25.969124 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:25.969106    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t"
Jul 31 02:13:25.971459 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:25.971436    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:13:25.976316 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:25.976294    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t"]
Jul 31 02:13:26.031213 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:26.031192    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3ac275-c6a5-431a-ba84-f6c0e905a6bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t\" (UID: \"9f3ac275-c6a5-431a-ba84-f6c0e905a6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t"
Jul 31 02:13:26.131681 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:26.131659    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3ac275-c6a5-431a-ba84-f6c0e905a6bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t\" (UID: \"9f3ac275-c6a5-431a-ba84-f6c0e905a6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t"
Jul 31 02:13:26.132227 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:26.132209    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3ac275-c6a5-431a-ba84-f6c0e905a6bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t\" (UID: \"9f3ac275-c6a5-431a-ba84-f6c0e905a6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t"
Jul 31 02:13:26.281904 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:26.281847    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t"
Jul 31 02:13:26.412474 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:26.412443    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t"]
Jul 31 02:13:26.413655 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:13:26.413626    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3ac275_c6a5_431a_ba84_f6c0e905a6bd.slice/crio-07004182109ba7ae054c704935e64371ad37cecac842be7f6de424b50edd4ef4 WatchSource:0}: Error finding container 07004182109ba7ae054c704935e64371ad37cecac842be7f6de424b50edd4ef4: Status 404 returned error can't find the container with id 07004182109ba7ae054c704935e64371ad37cecac842be7f6de424b50edd4ef4
Jul 31 02:13:26.725890 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:26.725855    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t" event={"ID":"9f3ac275-c6a5-431a-ba84-f6c0e905a6bd","Type":"ContainerStarted","Data":"49c1267d00b86415f650acc258f42d714899201623ec3cd52be2fc52a74da8b4"}
Jul 31 02:13:26.725982 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:26.725918    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t" event={"ID":"9f3ac275-c6a5-431a-ba84-f6c0e905a6bd","Type":"ContainerStarted","Data":"07004182109ba7ae054c704935e64371ad37cecac842be7f6de424b50edd4ef4"}
Jul 31 02:13:26.740810 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:26.740763    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t" podStartSLOduration=1.740748677 podStartE2EDuration="1.740748677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:13:26.738938774 +0000 UTC m=+2165.104684402" watchObservedRunningTime="2026-07-31 02:13:26.740748677 +0000 UTC m=+2165.106494305"
Jul 31 02:13:27.731669 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:27.731638    2556 generic.go:356] "Generic (PLEG): container finished" podID="9f3ac275-c6a5-431a-ba84-f6c0e905a6bd" containerID="49c1267d00b86415f650acc258f42d714899201623ec3cd52be2fc52a74da8b4" exitCode=0
Jul 31 02:13:27.732055 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:27.731691    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t" event={"ID":"9f3ac275-c6a5-431a-ba84-f6c0e905a6bd","Type":"ContainerDied","Data":"49c1267d00b86415f650acc258f42d714899201623ec3cd52be2fc52a74da8b4"}
Jul 31 02:13:28.873424 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:28.873398    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t"
Jul 31 02:13:28.955057 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:28.955031    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3ac275-c6a5-431a-ba84-f6c0e905a6bd-maas-api-service-ca\") pod \"9f3ac275-c6a5-431a-ba84-f6c0e905a6bd\" (UID: \"9f3ac275-c6a5-431a-ba84-f6c0e905a6bd\") "
Jul 31 02:13:28.955356 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:28.955333    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f3ac275-c6a5-431a-ba84-f6c0e905a6bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f3ac275-c6a5-431a-ba84-f6c0e905a6bd" (UID: "9f3ac275-c6a5-431a-ba84-f6c0e905a6bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:13:29.055857 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:29.055834    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3ac275-c6a5-431a-ba84-f6c0e905a6bd-maas-api-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:13:29.742433 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:29.742400    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t"
Jul 31 02:13:29.742640 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:29.742441    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t" event={"ID":"9f3ac275-c6a5-431a-ba84-f6c0e905a6bd","Type":"ContainerDied","Data":"07004182109ba7ae054c704935e64371ad37cecac842be7f6de424b50edd4ef4"}
Jul 31 02:13:29.742640 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:29.742513    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07004182109ba7ae054c704935e64371ad37cecac842be7f6de424b50edd4ef4"
Jul 31 02:13:37.478310 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.478280    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"]
Jul 31 02:13:37.478793 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.478470    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x" podUID="ba3379a7-1432-4ec0-b4ac-82abacb8dede" containerName="maas-api" containerID="cri-o://a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8" gracePeriod=30
Jul 31 02:13:37.499424 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.499389    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:37.523418 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.522654    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"]
Jul 31 02:13:37.523418 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.522864    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw" podUID="a0e5552b-a118-425e-a432-4433d99e4332" containerName="payload-processing" containerID="cri-o://0506fd23e560aab671492ce2dd80f3e3b112e878fb4a777c4cb175e806bdecaa" gracePeriod=30
Jul 31 02:13:37.529316 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.529259    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"]
Jul 31 02:13:37.529497 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.529475    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c" podUID="18492aad-17c8-4a21-8497-b565aba7a111" containerName="payload-pre-processing" containerID="cri-o://8d0b9358b0e90fec2776323b7c6d1b72a52866ce14f19372e9176ac92108a6b3" gracePeriod=30
Jul 31 02:13:37.563141 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.563102    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:37.623471 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.623436    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:37.653121 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.653081    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t"]
Jul 31 02:13:37.655620 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.655594    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb65761c-7zx5t"]
Jul 31 02:13:37.679700 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.679501    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:37.788246 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.788000    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:13:37.788392 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.788316    2556 generic.go:356] "Generic (PLEG): container finished" podID="18492aad-17c8-4a21-8497-b565aba7a111" containerID="8d0b9358b0e90fec2776323b7c6d1b72a52866ce14f19372e9176ac92108a6b3" exitCode=0
Jul 31 02:13:37.788492 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.788470    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c" event={"ID":"18492aad-17c8-4a21-8497-b565aba7a111","Type":"ContainerDied","Data":"8d0b9358b0e90fec2776323b7c6d1b72a52866ce14f19372e9176ac92108a6b3"}
Jul 31 02:13:37.790812 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.790579    2556 generic.go:356] "Generic (PLEG): container finished" podID="a0e5552b-a118-425e-a432-4433d99e4332" containerID="0506fd23e560aab671492ce2dd80f3e3b112e878fb4a777c4cb175e806bdecaa" exitCode=0
Jul 31 02:13:37.790812 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.790687    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw" event={"ID":"a0e5552b-a118-425e-a432-4433d99e4332","Type":"ContainerDied","Data":"0506fd23e560aab671492ce2dd80f3e3b112e878fb4a777c4cb175e806bdecaa"}
Jul 31 02:13:37.793020 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.792991    2556 generic.go:356] "Generic (PLEG): container finished" podID="ba3379a7-1432-4ec0-b4ac-82abacb8dede" containerID="a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8" exitCode=0
Jul 31 02:13:37.793191 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.793046    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x" event={"ID":"ba3379a7-1432-4ec0-b4ac-82abacb8dede","Type":"ContainerDied","Data":"a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8"}
Jul 31 02:13:37.793191 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.793083    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x" event={"ID":"ba3379a7-1432-4ec0-b4ac-82abacb8dede","Type":"ContainerDied","Data":"96229252cffe3ccc92b26d1c88f0eb37595cdb9cf5a98a29da7aab00233f87a5"}
Jul 31 02:13:37.793191 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.793111    2556 scope.go:117] "RemoveContainer" containerID="a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8"
Jul 31 02:13:37.793462 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.793277    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"
Jul 31 02:13:37.813862 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.813821    2556 scope.go:117] "RemoveContainer" containerID="a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8"
Jul 31 02:13:37.814100 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:13:37.814079    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8\": container with ID starting with a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8 not found: ID does not exist" containerID="a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8"
Jul 31 02:13:37.814203 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.814110    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8"} err="failed to get container status \"a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8\": rpc error: code = NotFound desc = could not find container \"a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8\": container with ID starting with a4167a9944227a6cec4e902ccb8b7755894bd4a8b1a53f7e0e03f2605cd7efb8 not found: ID does not exist"
Jul 31 02:13:37.817535 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.817512    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"
Jul 31 02:13:37.818442 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.818422    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:37.826544 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.826522    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"
Jul 31 02:13:37.920518 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.920488    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b4f8\" (UniqueName: \"kubernetes.io/projected/a0e5552b-a118-425e-a432-4433d99e4332-kube-api-access-8b4f8\") pod \"a0e5552b-a118-425e-a432-4433d99e4332\" (UID: \"a0e5552b-a118-425e-a432-4433d99e4332\") "
Jul 31 02:13:37.920726 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.920525    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3379a7-1432-4ec0-b4ac-82abacb8dede-openshift-service-ca\") pod \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") "
Jul 31 02:13:37.920726 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.920548    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18492aad-17c8-4a21-8497-b565aba7a111-plugins-config-volume\") pod \"18492aad-17c8-4a21-8497-b565aba7a111\" (UID: \"18492aad-17c8-4a21-8497-b565aba7a111\") "
Jul 31 02:13:37.920726 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.920637    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e5552b-a118-425e-a432-4433d99e4332-plugins-config-volume\") pod \"a0e5552b-a118-425e-a432-4433d99e4332\" (UID: \"a0e5552b-a118-425e-a432-4433d99e4332\") "
Jul 31 02:13:37.920899 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.920720    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5f5k\" (UniqueName: \"kubernetes.io/projected/ba3379a7-1432-4ec0-b4ac-82abacb8dede-kube-api-access-j5f5k\") pod \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") "
Jul 31 02:13:37.920899 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.920756    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba3379a7-1432-4ec0-b4ac-82abacb8dede-maas-api-tls\") pod \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\" (UID: \"ba3379a7-1432-4ec0-b4ac-82abacb8dede\") "
Jul 31 02:13:37.920899 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.920779    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9tkc\" (UniqueName: \"kubernetes.io/projected/18492aad-17c8-4a21-8497-b565aba7a111-kube-api-access-w9tkc\") pod \"18492aad-17c8-4a21-8497-b565aba7a111\" (UID: \"18492aad-17c8-4a21-8497-b565aba7a111\") "
Jul 31 02:13:37.921056 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.921019    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18492aad-17c8-4a21-8497-b565aba7a111-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18492aad-17c8-4a21-8497-b565aba7a111" (UID: "18492aad-17c8-4a21-8497-b565aba7a111"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:13:37.921056 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.921023    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3379a7-1432-4ec0-b4ac-82abacb8dede-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ba3379a7-1432-4ec0-b4ac-82abacb8dede" (UID: "ba3379a7-1432-4ec0-b4ac-82abacb8dede"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:13:37.921190 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.921168    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e5552b-a118-425e-a432-4433d99e4332-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0e5552b-a118-425e-a432-4433d99e4332" (UID: "a0e5552b-a118-425e-a432-4433d99e4332"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:13:37.923135 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.923093    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba3379a7-1432-4ec0-b4ac-82abacb8dede-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ba3379a7-1432-4ec0-b4ac-82abacb8dede" (UID: "ba3379a7-1432-4ec0-b4ac-82abacb8dede"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:13:37.923135 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.923120    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e5552b-a118-425e-a432-4433d99e4332-kube-api-access-8b4f8" (OuterVolumeSpecName: "kube-api-access-8b4f8") pod "a0e5552b-a118-425e-a432-4433d99e4332" (UID: "a0e5552b-a118-425e-a432-4433d99e4332"). InnerVolumeSpecName "kube-api-access-8b4f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:13:37.923287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.923164    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18492aad-17c8-4a21-8497-b565aba7a111-kube-api-access-w9tkc" (OuterVolumeSpecName: "kube-api-access-w9tkc") pod "18492aad-17c8-4a21-8497-b565aba7a111" (UID: "18492aad-17c8-4a21-8497-b565aba7a111"). InnerVolumeSpecName "kube-api-access-w9tkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:13:37.923287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:37.923231    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3379a7-1432-4ec0-b4ac-82abacb8dede-kube-api-access-j5f5k" (OuterVolumeSpecName: "kube-api-access-j5f5k") pod "ba3379a7-1432-4ec0-b4ac-82abacb8dede" (UID: "ba3379a7-1432-4ec0-b4ac-82abacb8dede"). InnerVolumeSpecName "kube-api-access-j5f5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:13:38.021623 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.021502    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5f5k\" (UniqueName: \"kubernetes.io/projected/ba3379a7-1432-4ec0-b4ac-82abacb8dede-kube-api-access-j5f5k\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:13:38.021623 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.021532    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba3379a7-1432-4ec0-b4ac-82abacb8dede-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:13:38.021623 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.021546    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9tkc\" (UniqueName: \"kubernetes.io/projected/18492aad-17c8-4a21-8497-b565aba7a111-kube-api-access-w9tkc\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:13:38.021623 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.021586    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b4f8\" (UniqueName: \"kubernetes.io/projected/a0e5552b-a118-425e-a432-4433d99e4332-kube-api-access-8b4f8\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:13:38.021623 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.021599    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3379a7-1432-4ec0-b4ac-82abacb8dede-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:13:38.021623 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.021612    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18492aad-17c8-4a21-8497-b565aba7a111-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:13:38.021623 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.021623    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e5552b-a118-425e-a432-4433d99e4332-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:13:38.065293 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.065265    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:38.113607 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.113582    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"]
Jul 31 02:13:38.117300 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.117275    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb65761c-758c6fcc49-wn85x"]
Jul 31 02:13:38.151285 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.151258    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f3ac275-c6a5-431a-ba84-f6c0e905a6bd" path="/var/lib/kubelet/pods/9f3ac275-c6a5-431a-ba84-f6c0e905a6bd/volumes"
Jul 31 02:13:38.151631 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.151617    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba3379a7-1432-4ec0-b4ac-82abacb8dede" path="/var/lib/kubelet/pods/ba3379a7-1432-4ec0-b4ac-82abacb8dede/volumes"
Jul 31 02:13:38.536865 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.536841    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:38.648519 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.648497    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:38.805202 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.805176    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"
Jul 31 02:13:38.805321 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.805204    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c" event={"ID":"18492aad-17c8-4a21-8497-b565aba7a111","Type":"ContainerDied","Data":"96bf0dcb20c407d7570ebb0c461ed11e434d29968359198f3e2570c406125233"}
Jul 31 02:13:38.805321 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.805295    2556 scope.go:117] "RemoveContainer" containerID="8d0b9358b0e90fec2776323b7c6d1b72a52866ce14f19372e9176ac92108a6b3"
Jul 31 02:13:38.806712 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.806690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw" event={"ID":"a0e5552b-a118-425e-a432-4433d99e4332","Type":"ContainerDied","Data":"c8a8f4cc21bd19634e54abc02fade0521849e930bf17a2e3bc2b257a6c128e67"}
Jul 31 02:13:38.806809 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.806745    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"
Jul 31 02:13:38.807811 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.807787    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:38.816263 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.816245    2556 scope.go:117] "RemoveContainer" containerID="0506fd23e560aab671492ce2dd80f3e3b112e878fb4a777c4cb175e806bdecaa"
Jul 31 02:13:38.827100 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.827081    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"]
Jul 31 02:13:38.832166 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.832143    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cb65761c-8ddd79c68-wbc4c"]
Jul 31 02:13:38.842751 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.842731    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"]
Jul 31 02:13:38.846765 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:38.846747    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cb65761c-88646458b-928tw"]
Jul 31 02:13:40.151213 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:40.151161    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18492aad-17c8-4a21-8497-b565aba7a111" path="/var/lib/kubelet/pods/18492aad-17c8-4a21-8497-b565aba7a111/volumes"
Jul 31 02:13:40.151791 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:40.151706    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e5552b-a118-425e-a432-4433d99e4332" path="/var/lib/kubelet/pods/a0e5552b-a118-425e-a432-4433d99e4332/volumes"
Jul 31 02:13:50.583992 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:50.583946    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:56.757157 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:56.757120    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:56.859057 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:56.859021    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:56.958285 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:56.958248    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:57.058721 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:57.058694    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:13:57.561440 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:13:57.561412    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:14:23.623833 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:14:23.623801    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:14:29.547547 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:14:29.547515    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:14:29.651996 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:14:29.651964    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:14:29.752533 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:14:29.752507    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:14:29.848456 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:14:29.848428    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:14:45.790024 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:14:45.789939    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:14:51.862461 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:14:51.862428    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:14:51.962214 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:14:51.962183    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:00.139884 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.139845    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757735-zfcjm"]
Jul 31 02:15:00.140361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.140143    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18492aad-17c8-4a21-8497-b565aba7a111" containerName="payload-pre-processing"
Jul 31 02:15:00.140361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.140153    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="18492aad-17c8-4a21-8497-b565aba7a111" containerName="payload-pre-processing"
Jul 31 02:15:00.140361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.140165    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e5552b-a118-425e-a432-4433d99e4332" containerName="payload-processing"
Jul 31 02:15:00.140361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.140170    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e5552b-a118-425e-a432-4433d99e4332" containerName="payload-processing"
Jul 31 02:15:00.140361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.140178    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3379a7-1432-4ec0-b4ac-82abacb8dede" containerName="maas-api"
Jul 31 02:15:00.140361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.140186    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3379a7-1432-4ec0-b4ac-82abacb8dede" containerName="maas-api"
Jul 31 02:15:00.140361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.140194    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f3ac275-c6a5-431a-ba84-f6c0e905a6bd" containerName="revoke-keys"
Jul 31 02:15:00.140361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.140198    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f3ac275-c6a5-431a-ba84-f6c0e905a6bd" containerName="revoke-keys"
Jul 31 02:15:00.140361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.140258    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3379a7-1432-4ec0-b4ac-82abacb8dede" containerName="maas-api"
Jul 31 02:15:00.140361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.140266    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f3ac275-c6a5-431a-ba84-f6c0e905a6bd" containerName="revoke-keys"
Jul 31 02:15:00.140361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.140272    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e5552b-a118-425e-a432-4433d99e4332" containerName="payload-processing"
Jul 31 02:15:00.140361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.140278    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="18492aad-17c8-4a21-8497-b565aba7a111" containerName="payload-pre-processing"
Jul 31 02:15:00.143139 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.143124    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757735-zfcjm"
Jul 31 02:15:00.145387 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.145364    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:15:00.152526 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.152506    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757735-zfcjm"]
Jul 31 02:15:00.153851 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.153830    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757735-zfcjm"
Jul 31 02:15:00.282726 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:00.282693    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757735-zfcjm"]
Jul 31 02:15:00.283011 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:15:00.282981    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74e4016_055d_48be_8dca_a8e7e0d024fb.slice/crio-50a1702c692b201799227cf09d71e40e0d5838a5f7003e496ea4533616862ada WatchSource:0}: Error finding container 50a1702c692b201799227cf09d71e40e0d5838a5f7003e496ea4533616862ada: Status 404 returned error can't find the container with id 50a1702c692b201799227cf09d71e40e0d5838a5f7003e496ea4533616862ada
Jul 31 02:15:01.165980 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:01.165944    2556 generic.go:356] "Generic (PLEG): container finished" podID="e74e4016-055d-48be-8dca-a8e7e0d024fb" containerID="07fa93232a17623e18512a00ead030078cf2ed3f45ede43021482d9b50554de7" exitCode=0
Jul 31 02:15:01.166407 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:01.166027    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757735-zfcjm" event={"ID":"e74e4016-055d-48be-8dca-a8e7e0d024fb","Type":"ContainerDied","Data":"07fa93232a17623e18512a00ead030078cf2ed3f45ede43021482d9b50554de7"}
Jul 31 02:15:01.166407 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:01.166078    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757735-zfcjm" event={"ID":"e74e4016-055d-48be-8dca-a8e7e0d024fb","Type":"ContainerStarted","Data":"50a1702c692b201799227cf09d71e40e0d5838a5f7003e496ea4533616862ada"}
Jul 31 02:15:02.298795 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:02.298768    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757735-zfcjm"
Jul 31 02:15:02.368260 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:02.368224    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:03.175787 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:03.175756    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757735-zfcjm" event={"ID":"e74e4016-055d-48be-8dca-a8e7e0d024fb","Type":"ContainerDied","Data":"50a1702c692b201799227cf09d71e40e0d5838a5f7003e496ea4533616862ada"}
Jul 31 02:15:03.175787 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:03.175773    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757735-zfcjm"
Jul 31 02:15:03.175999 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:03.175804    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50a1702c692b201799227cf09d71e40e0d5838a5f7003e496ea4533616862ada"
Jul 31 02:15:13.522254 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:13.522216    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:19.669650 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:19.669603    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:19.773773 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:19.773741    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:19.869664 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:19.869631    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:19.986486 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:19.986398    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:20.758035 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:20.758000    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:20.872344 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:20.872311    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:21.010733 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:21.010629    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:22.841319 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:22.841283    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:28.759647 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:28.759612    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:28.863420 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:28.863385    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:29.515832 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:29.515795    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:35.376850 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:35.376814    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:35.477416 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:35.477383    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:35.657628 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:35.657550    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:35.969644 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:35.969549    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:36.448497 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:36.448464    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:39.462263 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:39.462222    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:39.562488 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:39.562454    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:40.761637 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.761607    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"]
Jul 31 02:15:40.761997 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.761920    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74e4016-055d-48be-8dca-a8e7e0d024fb" containerName="cleanup"
Jul 31 02:15:40.761997 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.761931    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74e4016-055d-48be-8dca-a8e7e0d024fb" containerName="cleanup"
Jul 31 02:15:40.761997 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.761977    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74e4016-055d-48be-8dca-a8e7e0d024fb" containerName="cleanup"
Jul 31 02:15:40.764944 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.764926    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:40.768158 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.768138    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f714090a-serving-cert\""
Jul 31 02:15:40.788002 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.787972    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"]
Jul 31 02:15:40.794381 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.794349    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sj45\" (UniqueName: \"kubernetes.io/projected/ae311f75-d083-4469-8825-1af5c4674a18-kube-api-access-2sj45\") pod \"maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:40.794381 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.794381    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae311f75-d083-4469-8825-1af5c4674a18-maas-api-tls\") pod \"maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:40.794594 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.794407    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae311f75-d083-4469-8825-1af5c4674a18-openshift-service-ca\") pod \"maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:40.821139 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.821111    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:40.895498 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.895463    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sj45\" (UniqueName: \"kubernetes.io/projected/ae311f75-d083-4469-8825-1af5c4674a18-kube-api-access-2sj45\") pod \"maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:40.895679 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.895501    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae311f75-d083-4469-8825-1af5c4674a18-maas-api-tls\") pod \"maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:40.895679 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.895539    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae311f75-d083-4469-8825-1af5c4674a18-openshift-service-ca\") pod \"maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:40.895679 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:15:40.895661    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-serving-cert: secret "maas-api-e2e-mt-f714090a-serving-cert" not found
Jul 31 02:15:40.895799 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:15:40.895734    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae311f75-d083-4469-8825-1af5c4674a18-maas-api-tls podName:ae311f75-d083-4469-8825-1af5c4674a18 nodeName:}" failed. No retries permitted until 2026-07-31 02:15:41.395717724 +0000 UTC m=+2299.761463329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae311f75-d083-4469-8825-1af5c4674a18-maas-api-tls") pod "maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m" (UID: "ae311f75-d083-4469-8825-1af5c4674a18") : secret "maas-api-e2e-mt-f714090a-serving-cert" not found
Jul 31 02:15:40.896141 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.896123    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae311f75-d083-4469-8825-1af5c4674a18-openshift-service-ca\") pod \"maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:40.905086 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:40.905057    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sj45\" (UniqueName: \"kubernetes.io/projected/ae311f75-d083-4469-8825-1af5c4674a18-kube-api-access-2sj45\") pod \"maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:41.263496 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.263454    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:41.358109 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.358077    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:41.364628 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.364604    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"]
Jul 31 02:15:41.367974 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.367953    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"
Jul 31 02:15:41.370533 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.370509    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f714090a-dockercfg-9gcjs\""
Jul 31 02:15:41.370661 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.370607    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f714090a\""
Jul 31 02:15:41.381043 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.381020    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"]
Jul 31 02:15:41.398756 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.398731    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c32654-86aa-45a5-b08c-e9b39dcc2090-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv\" (UID: \"b7c32654-86aa-45a5-b08c-e9b39dcc2090\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"
Jul 31 02:15:41.398878 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.398765    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84mk2\" (UniqueName: \"kubernetes.io/projected/b7c32654-86aa-45a5-b08c-e9b39dcc2090-kube-api-access-84mk2\") pod \"payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv\" (UID: \"b7c32654-86aa-45a5-b08c-e9b39dcc2090\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"
Jul 31 02:15:41.398878 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.398808    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"]
Jul 31 02:15:41.398989 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.398811    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae311f75-d083-4469-8825-1af5c4674a18-maas-api-tls\") pod \"maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:41.401155 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.401135    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae311f75-d083-4469-8825-1af5c4674a18-maas-api-tls\") pod \"maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:41.402580 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.402545    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"
Jul 31 02:15:41.408952 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.408929    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"]
Jul 31 02:15:41.499379 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.499342    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fda7f1-3817-4873-95cb-db22a69b4e24-plugins-config-volume\") pod \"payload-processing-e2e-mt-f714090a-76746f556b-f9z9q\" (UID: \"52fda7f1-3817-4873-95cb-db22a69b4e24\") " pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"
Jul 31 02:15:41.499612 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.499404    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq85h\" (UniqueName: \"kubernetes.io/projected/52fda7f1-3817-4873-95cb-db22a69b4e24-kube-api-access-hq85h\") pod \"payload-processing-e2e-mt-f714090a-76746f556b-f9z9q\" (UID: \"52fda7f1-3817-4873-95cb-db22a69b4e24\") " pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"
Jul 31 02:15:41.499612 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.499461    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c32654-86aa-45a5-b08c-e9b39dcc2090-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv\" (UID: \"b7c32654-86aa-45a5-b08c-e9b39dcc2090\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"
Jul 31 02:15:41.499612 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.499491    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84mk2\" (UniqueName: \"kubernetes.io/projected/b7c32654-86aa-45a5-b08c-e9b39dcc2090-kube-api-access-84mk2\") pod \"payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv\" (UID: \"b7c32654-86aa-45a5-b08c-e9b39dcc2090\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"
Jul 31 02:15:41.500217 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.500193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c32654-86aa-45a5-b08c-e9b39dcc2090-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv\" (UID: \"b7c32654-86aa-45a5-b08c-e9b39dcc2090\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"
Jul 31 02:15:41.509266 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.509241    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:15:41.510574 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.510542    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84mk2\" (UniqueName: \"kubernetes.io/projected/b7c32654-86aa-45a5-b08c-e9b39dcc2090-kube-api-access-84mk2\") pod \"payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv\" (UID: \"b7c32654-86aa-45a5-b08c-e9b39dcc2090\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"
Jul 31 02:15:41.600636 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.600608    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fda7f1-3817-4873-95cb-db22a69b4e24-plugins-config-volume\") pod \"payload-processing-e2e-mt-f714090a-76746f556b-f9z9q\" (UID: \"52fda7f1-3817-4873-95cb-db22a69b4e24\") " pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"
Jul 31 02:15:41.600839 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.600652    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq85h\" (UniqueName: \"kubernetes.io/projected/52fda7f1-3817-4873-95cb-db22a69b4e24-kube-api-access-hq85h\") pod \"payload-processing-e2e-mt-f714090a-76746f556b-f9z9q\" (UID: \"52fda7f1-3817-4873-95cb-db22a69b4e24\") " pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"
Jul 31 02:15:41.601325 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.601302    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fda7f1-3817-4873-95cb-db22a69b4e24-plugins-config-volume\") pod \"payload-processing-e2e-mt-f714090a-76746f556b-f9z9q\" (UID: \"52fda7f1-3817-4873-95cb-db22a69b4e24\") " pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"
Jul 31 02:15:41.607918 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.607894    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq85h\" (UniqueName: \"kubernetes.io/projected/52fda7f1-3817-4873-95cb-db22a69b4e24-kube-api-access-hq85h\") pod \"payload-processing-e2e-mt-f714090a-76746f556b-f9z9q\" (UID: \"52fda7f1-3817-4873-95cb-db22a69b4e24\") " pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"
Jul 31 02:15:41.674390 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.674359    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:41.678545 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.678522    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"
Jul 31 02:15:41.717644 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.717608    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"
Jul 31 02:15:41.833911 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.833785    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"]
Jul 31 02:15:41.837714 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:15:41.837683    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae311f75_d083_4469_8825_1af5c4674a18.slice/crio-152215a5df52c519a16ffa0156d94ea6596941f9b7765dc78028cac836a240c0 WatchSource:0}: Error finding container 152215a5df52c519a16ffa0156d94ea6596941f9b7765dc78028cac836a240c0: Status 404 returned error can't find the container with id 152215a5df52c519a16ffa0156d94ea6596941f9b7765dc78028cac836a240c0
Jul 31 02:15:41.853213 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.853187    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"]
Jul 31 02:15:41.882102 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:41.882080    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"]
Jul 31 02:15:41.890672 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:15:41.890640    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fda7f1_3817_4873_95cb_db22a69b4e24.slice/crio-3dc4216b5e571a45a09cfd7e6f5beff3165fb6c39eef7cb8662c2bf1d881028e WatchSource:0}: Error finding container 3dc4216b5e571a45a09cfd7e6f5beff3165fb6c39eef7cb8662c2bf1d881028e: Status 404 returned error can't find the container with id 3dc4216b5e571a45a09cfd7e6f5beff3165fb6c39eef7cb8662c2bf1d881028e
Jul 31 02:15:42.342912 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:42.342870    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q" event={"ID":"52fda7f1-3817-4873-95cb-db22a69b4e24","Type":"ContainerStarted","Data":"4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430"}
Jul 31 02:15:42.343124 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:42.342943    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"
Jul 31 02:15:42.343124 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:42.342965    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q" event={"ID":"52fda7f1-3817-4873-95cb-db22a69b4e24","Type":"ContainerStarted","Data":"3dc4216b5e571a45a09cfd7e6f5beff3165fb6c39eef7cb8662c2bf1d881028e"}
Jul 31 02:15:42.344298 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:42.344272    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m" event={"ID":"ae311f75-d083-4469-8825-1af5c4674a18","Type":"ContainerStarted","Data":"98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb"}
Jul 31 02:15:42.344416 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:42.344316    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m" event={"ID":"ae311f75-d083-4469-8825-1af5c4674a18","Type":"ContainerStarted","Data":"152215a5df52c519a16ffa0156d94ea6596941f9b7765dc78028cac836a240c0"}
Jul 31 02:15:42.344416 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:42.344336    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:42.345698 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:42.345678    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv" event={"ID":"b7c32654-86aa-45a5-b08c-e9b39dcc2090","Type":"ContainerStarted","Data":"94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc"}
Jul 31 02:15:42.345845 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:42.345712    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv" event={"ID":"b7c32654-86aa-45a5-b08c-e9b39dcc2090","Type":"ContainerStarted","Data":"11812772659ac52cb9641b4e245f9ee377a69d635a79303f4eabe933fb30ffa0"}
Jul 31 02:15:42.345845 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:42.345744    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"
Jul 31 02:15:42.358873 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:42.358830    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q" podStartSLOduration=1.358815511 podStartE2EDuration="1.358815511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:15:42.357300488 +0000 UTC m=+2300.723046114" watchObservedRunningTime="2026-07-31 02:15:42.358815511 +0000 UTC m=+2300.724561116"
Jul 31 02:15:42.370699 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:42.370646    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv" podStartSLOduration=1.370627225 podStartE2EDuration="1.370627225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:15:42.370617599 +0000 UTC m=+2300.736363227" watchObservedRunningTime="2026-07-31 02:15:42.370627225 +0000 UTC m=+2300.736372851"
Jul 31 02:15:42.385863 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:42.385807    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m" podStartSLOduration=1.980405618 podStartE2EDuration="2.385788083s" totalImagesPullingTime="405.382465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:15:42.384528042 +0000 UTC m=+2300.750273663" watchObservedRunningTime="2026-07-31 02:15:42.385788083 +0000 UTC m=+2300.751533711"
Jul 31 02:15:48.356212 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:48.356185    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:15:53.352740 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:53.352709    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"
Jul 31 02:15:53.353169 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:15:53.353034    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"
Jul 31 02:16:04.554649 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:04.554610    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:05.235962 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:05.235926    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:05.337051 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:05.337010    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:05.737794 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:05.737762    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:11.345251 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:11.345220    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:11.715985 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:11.715907    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m"]
Jul 31 02:16:11.719356 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:11.719341    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m"
Jul 31 02:16:11.721492 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:11.721474    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:16:11.726973 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:11.726948    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m"]
Jul 31 02:16:11.854973 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:11.854946    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f333fdc1-9361-4901-be81-1167ab6d088e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f714090a-b5x6m\" (UID: \"f333fdc1-9361-4901-be81-1167ab6d088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m"
Jul 31 02:16:11.960928 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:11.955879    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f333fdc1-9361-4901-be81-1167ab6d088e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f714090a-b5x6m\" (UID: \"f333fdc1-9361-4901-be81-1167ab6d088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m"
Jul 31 02:16:11.960928 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:11.956610    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f333fdc1-9361-4901-be81-1167ab6d088e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f714090a-b5x6m\" (UID: \"f333fdc1-9361-4901-be81-1167ab6d088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m"
Jul 31 02:16:12.029687 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:12.029633    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m"
Jul 31 02:16:12.154126 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:12.154100    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m"]
Jul 31 02:16:12.157487 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:16:12.157459    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf333fdc1_9361_4901_be81_1167ab6d088e.slice/crio-46d87e23b1828ab492b604be024f1fe3a78ff3e4bc55b9a7316c15c737c18c8f WatchSource:0}: Error finding container 46d87e23b1828ab492b604be024f1fe3a78ff3e4bc55b9a7316c15c737c18c8f: Status 404 returned error can't find the container with id 46d87e23b1828ab492b604be024f1fe3a78ff3e4bc55b9a7316c15c737c18c8f
Jul 31 02:16:12.481495 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:12.481468    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m" event={"ID":"f333fdc1-9361-4901-be81-1167ab6d088e","Type":"ContainerStarted","Data":"46d87e23b1828ab492b604be024f1fe3a78ff3e4bc55b9a7316c15c737c18c8f"}
Jul 31 02:16:12.687120 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:12.687038    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:13.487580 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:13.487533    2556 generic.go:356] "Generic (PLEG): container finished" podID="f333fdc1-9361-4901-be81-1167ab6d088e" containerID="c24ceeb4c9f9b174b64dba7366db943abfd89467387b9a5023dee88f0754ea13" exitCode=0
Jul 31 02:16:13.487950 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:13.487630    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m" event={"ID":"f333fdc1-9361-4901-be81-1167ab6d088e","Type":"ContainerDied","Data":"c24ceeb4c9f9b174b64dba7366db943abfd89467387b9a5023dee88f0754ea13"}
Jul 31 02:16:14.614908 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:14.614886    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m"
Jul 31 02:16:14.778537 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:14.778469    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f333fdc1-9361-4901-be81-1167ab6d088e-maas-api-service-ca\") pod \"f333fdc1-9361-4901-be81-1167ab6d088e\" (UID: \"f333fdc1-9361-4901-be81-1167ab6d088e\") "
Jul 31 02:16:14.778824 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:14.778802    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f333fdc1-9361-4901-be81-1167ab6d088e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f333fdc1-9361-4901-be81-1167ab6d088e" (UID: "f333fdc1-9361-4901-be81-1167ab6d088e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:16:14.879467 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:14.879438    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f333fdc1-9361-4901-be81-1167ab6d088e-maas-api-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:16:15.497616 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:15.497595    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m"
Jul 31 02:16:15.497751 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:15.497626    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m" event={"ID":"f333fdc1-9361-4901-be81-1167ab6d088e","Type":"ContainerDied","Data":"46d87e23b1828ab492b604be024f1fe3a78ff3e4bc55b9a7316c15c737c18c8f"}
Jul 31 02:16:15.497751 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:15.497673    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46d87e23b1828ab492b604be024f1fe3a78ff3e4bc55b9a7316c15c737c18c8f"
Jul 31 02:16:22.073011 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.072976    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"]
Jul 31 02:16:22.073427 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.073248    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m" podUID="ae311f75-d083-4469-8825-1af5c4674a18" containerName="maas-api" containerID="cri-o://98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb" gracePeriod=30
Jul 31 02:16:22.097881 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.097856    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:22.133131 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.133107    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"]
Jul 31 02:16:22.133303 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.133275    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q" podUID="52fda7f1-3817-4873-95cb-db22a69b4e24" containerName="payload-processing" containerID="cri-o://4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430" gracePeriod=30
Jul 31 02:16:22.140315 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.140292    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"]
Jul 31 02:16:22.140529 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.140501    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv" podUID="b7c32654-86aa-45a5-b08c-e9b39dcc2090" containerName="payload-pre-processing" containerID="cri-o://94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc" gracePeriod=30
Jul 31 02:16:22.173131 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.173107    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:22.235463 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.235439    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:22.269715 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.269678    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m"]
Jul 31 02:16:22.274718 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.274690    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f714090a-b5x6m"]
Jul 31 02:16:22.298932 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.298902    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:22.399849 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.399827    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:16:22.401824 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.401807    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"
Jul 31 02:16:22.428522 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.428498    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"
Jul 31 02:16:22.529672 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.529642    2556 generic.go:356] "Generic (PLEG): container finished" podID="52fda7f1-3817-4873-95cb-db22a69b4e24" containerID="4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430" exitCode=0
Jul 31 02:16:22.529812 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.529679    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"
Jul 31 02:16:22.529812 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.529726    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q" event={"ID":"52fda7f1-3817-4873-95cb-db22a69b4e24","Type":"ContainerDied","Data":"4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430"}
Jul 31 02:16:22.529812 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.529807    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q" event={"ID":"52fda7f1-3817-4873-95cb-db22a69b4e24","Type":"ContainerDied","Data":"3dc4216b5e571a45a09cfd7e6f5beff3165fb6c39eef7cb8662c2bf1d881028e"}
Jul 31 02:16:22.529953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.529829    2556 scope.go:117] "RemoveContainer" containerID="4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430"
Jul 31 02:16:22.531063 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.531036    2556 generic.go:356] "Generic (PLEG): container finished" podID="ae311f75-d083-4469-8825-1af5c4674a18" containerID="98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb" exitCode=0
Jul 31 02:16:22.531179 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.531061    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m" event={"ID":"ae311f75-d083-4469-8825-1af5c4674a18","Type":"ContainerDied","Data":"98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb"}
Jul 31 02:16:22.531179 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.531075    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"
Jul 31 02:16:22.531179 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.531106    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m" event={"ID":"ae311f75-d083-4469-8825-1af5c4674a18","Type":"ContainerDied","Data":"152215a5df52c519a16ffa0156d94ea6596941f9b7765dc78028cac836a240c0"}
Jul 31 02:16:22.532353 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.532328    2556 generic.go:356] "Generic (PLEG): container finished" podID="b7c32654-86aa-45a5-b08c-e9b39dcc2090" containerID="94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc" exitCode=0
Jul 31 02:16:22.532471 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.532355    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv" event={"ID":"b7c32654-86aa-45a5-b08c-e9b39dcc2090","Type":"ContainerDied","Data":"94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc"}
Jul 31 02:16:22.532471 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.532398    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv" event={"ID":"b7c32654-86aa-45a5-b08c-e9b39dcc2090","Type":"ContainerDied","Data":"11812772659ac52cb9641b4e245f9ee377a69d635a79303f4eabe933fb30ffa0"}
Jul 31 02:16:22.532471 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.532362    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"
Jul 31 02:16:22.535331 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.535312    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq85h\" (UniqueName: \"kubernetes.io/projected/52fda7f1-3817-4873-95cb-db22a69b4e24-kube-api-access-hq85h\") pod \"52fda7f1-3817-4873-95cb-db22a69b4e24\" (UID: \"52fda7f1-3817-4873-95cb-db22a69b4e24\") "
Jul 31 02:16:22.535429 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.535338    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sj45\" (UniqueName: \"kubernetes.io/projected/ae311f75-d083-4469-8825-1af5c4674a18-kube-api-access-2sj45\") pod \"ae311f75-d083-4469-8825-1af5c4674a18\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") "
Jul 31 02:16:22.535429 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.535379    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c32654-86aa-45a5-b08c-e9b39dcc2090-plugins-config-volume\") pod \"b7c32654-86aa-45a5-b08c-e9b39dcc2090\" (UID: \"b7c32654-86aa-45a5-b08c-e9b39dcc2090\") "
Jul 31 02:16:22.535429 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.535394    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84mk2\" (UniqueName: \"kubernetes.io/projected/b7c32654-86aa-45a5-b08c-e9b39dcc2090-kube-api-access-84mk2\") pod \"b7c32654-86aa-45a5-b08c-e9b39dcc2090\" (UID: \"b7c32654-86aa-45a5-b08c-e9b39dcc2090\") "
Jul 31 02:16:22.535429 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.535425    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae311f75-d083-4469-8825-1af5c4674a18-openshift-service-ca\") pod \"ae311f75-d083-4469-8825-1af5c4674a18\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") "
Jul 31 02:16:22.535678 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.535446    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae311f75-d083-4469-8825-1af5c4674a18-maas-api-tls\") pod \"ae311f75-d083-4469-8825-1af5c4674a18\" (UID: \"ae311f75-d083-4469-8825-1af5c4674a18\") "
Jul 31 02:16:22.535678 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.535461    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fda7f1-3817-4873-95cb-db22a69b4e24-plugins-config-volume\") pod \"52fda7f1-3817-4873-95cb-db22a69b4e24\" (UID: \"52fda7f1-3817-4873-95cb-db22a69b4e24\") "
Jul 31 02:16:22.536066 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.535843    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c32654-86aa-45a5-b08c-e9b39dcc2090-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7c32654-86aa-45a5-b08c-e9b39dcc2090" (UID: "b7c32654-86aa-45a5-b08c-e9b39dcc2090"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:16:22.536066 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.535867    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae311f75-d083-4469-8825-1af5c4674a18-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae311f75-d083-4469-8825-1af5c4674a18" (UID: "ae311f75-d083-4469-8825-1af5c4674a18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:16:22.536066 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.536046    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52fda7f1-3817-4873-95cb-db22a69b4e24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "52fda7f1-3817-4873-95cb-db22a69b4e24" (UID: "52fda7f1-3817-4873-95cb-db22a69b4e24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:16:22.537595 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.537546    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae311f75-d083-4469-8825-1af5c4674a18-kube-api-access-2sj45" (OuterVolumeSpecName: "kube-api-access-2sj45") pod "ae311f75-d083-4469-8825-1af5c4674a18" (UID: "ae311f75-d083-4469-8825-1af5c4674a18"). InnerVolumeSpecName "kube-api-access-2sj45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:16:22.537729 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.537704    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae311f75-d083-4469-8825-1af5c4674a18-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae311f75-d083-4469-8825-1af5c4674a18" (UID: "ae311f75-d083-4469-8825-1af5c4674a18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:16:22.538044 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.538015    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fda7f1-3817-4873-95cb-db22a69b4e24-kube-api-access-hq85h" (OuterVolumeSpecName: "kube-api-access-hq85h") pod "52fda7f1-3817-4873-95cb-db22a69b4e24" (UID: "52fda7f1-3817-4873-95cb-db22a69b4e24"). InnerVolumeSpecName "kube-api-access-hq85h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:16:22.538133 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.538036    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c32654-86aa-45a5-b08c-e9b39dcc2090-kube-api-access-84mk2" (OuterVolumeSpecName: "kube-api-access-84mk2") pod "b7c32654-86aa-45a5-b08c-e9b39dcc2090" (UID: "b7c32654-86aa-45a5-b08c-e9b39dcc2090"). InnerVolumeSpecName "kube-api-access-84mk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:16:22.539178 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.539162    2556 scope.go:117] "RemoveContainer" containerID="4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430"
Jul 31 02:16:22.539458 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:16:22.539440    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430\": container with ID starting with 4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430 not found: ID does not exist" containerID="4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430"
Jul 31 02:16:22.539514 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.539466    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430"} err="failed to get container status \"4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430\": rpc error: code = NotFound desc = could not find container \"4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430\": container with ID starting with 4fabdbd2b37fd27ffbceacba8f2aeb699a924ac7e16cf6160680a98563f2f430 not found: ID does not exist"
Jul 31 02:16:22.539514 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.539484    2556 scope.go:117] "RemoveContainer" containerID="98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb"
Jul 31 02:16:22.546666 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.546645    2556 scope.go:117] "RemoveContainer" containerID="98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb"
Jul 31 02:16:22.546906 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:16:22.546887    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb\": container with ID starting with 98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb not found: ID does not exist" containerID="98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb"
Jul 31 02:16:22.546959 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.546913    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb"} err="failed to get container status \"98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb\": rpc error: code = NotFound desc = could not find container \"98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb\": container with ID starting with 98bdee89a91f664cd9daa5e95d1f74c747cb4ccbb62dbced08941979bfe374cb not found: ID does not exist"
Jul 31 02:16:22.546959 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.546929    2556 scope.go:117] "RemoveContainer" containerID="94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc"
Jul 31 02:16:22.554017 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.553999    2556 scope.go:117] "RemoveContainer" containerID="94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc"
Jul 31 02:16:22.554249 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:16:22.554230    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc\": container with ID starting with 94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc not found: ID does not exist" containerID="94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc"
Jul 31 02:16:22.554307 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.554253    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc"} err="failed to get container status \"94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc\": rpc error: code = NotFound desc = could not find container \"94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc\": container with ID starting with 94c14c87a452a647d7fb22b1ada1d7c48d4919e4c3b0edac0e9263bbe461c9fc not found: ID does not exist"
Jul 31 02:16:22.636402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.636352    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c32654-86aa-45a5-b08c-e9b39dcc2090-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:16:22.636402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.636370    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84mk2\" (UniqueName: \"kubernetes.io/projected/b7c32654-86aa-45a5-b08c-e9b39dcc2090-kube-api-access-84mk2\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:16:22.636402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.636379    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae311f75-d083-4469-8825-1af5c4674a18-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:16:22.636402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.636389    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae311f75-d083-4469-8825-1af5c4674a18-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:16:22.636402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.636398    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fda7f1-3817-4873-95cb-db22a69b4e24-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:16:22.636635 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.636405    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq85h\" (UniqueName: \"kubernetes.io/projected/52fda7f1-3817-4873-95cb-db22a69b4e24-kube-api-access-hq85h\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:16:22.636635 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.636415    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sj45\" (UniqueName: \"kubernetes.io/projected/ae311f75-d083-4469-8825-1af5c4674a18-kube-api-access-2sj45\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:16:22.645758 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.645742    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:22.861966 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.861944    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"]
Jul 31 02:16:22.865674 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.865656    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f714090a-76746f556b-f9z9q"]
Jul 31 02:16:22.874846 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.874828    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"]
Jul 31 02:16:22.878742 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.878724    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f714090a-6dd5bc66dd-xpnkv"]
Jul 31 02:16:22.888175 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.888132    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"]
Jul 31 02:16:22.894191 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:22.894170    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f714090a-56c6bdfb9b-ftz5m"]
Jul 31 02:16:24.150173 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:24.150131    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52fda7f1-3817-4873-95cb-db22a69b4e24" path="/var/lib/kubelet/pods/52fda7f1-3817-4873-95cb-db22a69b4e24/volumes"
Jul 31 02:16:24.150651 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:24.150636    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae311f75-d083-4469-8825-1af5c4674a18" path="/var/lib/kubelet/pods/ae311f75-d083-4469-8825-1af5c4674a18/volumes"
Jul 31 02:16:24.150981 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:24.150962    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c32654-86aa-45a5-b08c-e9b39dcc2090" path="/var/lib/kubelet/pods/b7c32654-86aa-45a5-b08c-e9b39dcc2090/volumes"
Jul 31 02:16:24.151283 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:24.151266    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f333fdc1-9361-4901-be81-1167ab6d088e" path="/var/lib/kubelet/pods/f333fdc1-9361-4901-be81-1167ab6d088e/volumes"
Jul 31 02:16:28.530365 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:28.530333    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:28.592046 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:28.592021    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:28.811707 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:28.811677    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:29.052192 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:29.052164    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:29.180629 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:29.180502    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:29.281265 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:29.281239    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:29.406682 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:29.406659    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:35.084326 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:35.084292    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:35.194347 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:35.194318    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:35.284807 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:35.284781    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:35.434343 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:35.434284    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:40.967794 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:40.967763    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:41.068123 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:41.068101    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:16:48.349154 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:16:48.349117    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:18:08.311678 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:08.311588    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:18:22.768337 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:22.768307    2556 scope.go:117] "RemoveContainer" containerID="ece540d92205e277c4fd1deadb7eaccd4c95053b2b8ced12feb82e11b358c311"
Jul 31 02:18:22.775690 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:22.775669    2556 scope.go:117] "RemoveContainer" containerID="afd9d035a510f57d28e1c3f9ddbfd1124f8121da60fc3edd658a578c246f9c0c"
Jul 31 02:18:40.723241 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:40.723204    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:18:43.879545 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:43.879515    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:18:44.270128 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:44.270059    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:18:44.358667 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:44.358631    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:18:47.409667 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:47.409627    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:18:47.508963 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:47.508923    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:18:48.493423 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.493395    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"]
Jul 31 02:18:48.493892 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.493873    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52fda7f1-3817-4873-95cb-db22a69b4e24" containerName="payload-processing"
Jul 31 02:18:48.493967 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.493894    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="52fda7f1-3817-4873-95cb-db22a69b4e24" containerName="payload-processing"
Jul 31 02:18:48.493967 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.493920    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c32654-86aa-45a5-b08c-e9b39dcc2090" containerName="payload-pre-processing"
Jul 31 02:18:48.493967 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.493929    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c32654-86aa-45a5-b08c-e9b39dcc2090" containerName="payload-pre-processing"
Jul 31 02:18:48.493967 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.493950    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f333fdc1-9361-4901-be81-1167ab6d088e" containerName="revoke-keys"
Jul 31 02:18:48.493967 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.493959    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f333fdc1-9361-4901-be81-1167ab6d088e" containerName="revoke-keys"
Jul 31 02:18:48.494245 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.493980    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae311f75-d083-4469-8825-1af5c4674a18" containerName="maas-api"
Jul 31 02:18:48.494245 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.493988    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae311f75-d083-4469-8825-1af5c4674a18" containerName="maas-api"
Jul 31 02:18:48.494245 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.494058    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c32654-86aa-45a5-b08c-e9b39dcc2090" containerName="payload-pre-processing"
Jul 31 02:18:48.494245 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.494070    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f333fdc1-9361-4901-be81-1167ab6d088e" containerName="revoke-keys"
Jul 31 02:18:48.494245 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.494079    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae311f75-d083-4469-8825-1af5c4674a18" containerName="maas-api"
Jul 31 02:18:48.494245 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.494089    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="52fda7f1-3817-4873-95cb-db22a69b4e24" containerName="payload-processing"
Jul 31 02:18:48.497192 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.497173    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:48.499551 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.499532    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a1d99bab-serving-cert\""
Jul 31 02:18:48.509009 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.508986    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"]
Jul 31 02:18:48.529727 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.529703    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:18:48.556060 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.556035    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw7hl\" (UniqueName: \"kubernetes.io/projected/3aee28f4-06aa-4ede-a75e-d9924b729239-kube-api-access-fw7hl\") pod \"maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:48.556161 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.556103    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aee28f4-06aa-4ede-a75e-d9924b729239-openshift-service-ca\") pod \"maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:48.556161 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.556157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aee28f4-06aa-4ede-a75e-d9924b729239-maas-api-tls\") pod \"maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:48.657470 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.657449    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aee28f4-06aa-4ede-a75e-d9924b729239-openshift-service-ca\") pod \"maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:48.657602 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.657485    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aee28f4-06aa-4ede-a75e-d9924b729239-maas-api-tls\") pod \"maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:48.657602 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.657506    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7hl\" (UniqueName: \"kubernetes.io/projected/3aee28f4-06aa-4ede-a75e-d9924b729239-kube-api-access-fw7hl\") pod \"maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:48.657723 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:18:48.657639    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-serving-cert: secret "maas-api-e2e-mt-a1d99bab-serving-cert" not found
Jul 31 02:18:48.657723 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:18:48.657697    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aee28f4-06aa-4ede-a75e-d9924b729239-maas-api-tls podName:3aee28f4-06aa-4ede-a75e-d9924b729239 nodeName:}" failed. No retries permitted until 2026-07-31 02:18:49.157681063 +0000 UTC m=+2487.523426668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3aee28f4-06aa-4ede-a75e-d9924b729239-maas-api-tls") pod "maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr" (UID: "3aee28f4-06aa-4ede-a75e-d9924b729239") : secret "maas-api-e2e-mt-a1d99bab-serving-cert" not found
Jul 31 02:18:48.658119 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.658096    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aee28f4-06aa-4ede-a75e-d9924b729239-openshift-service-ca\") pod \"maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:48.665143 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.665124    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7hl\" (UniqueName: \"kubernetes.io/projected/3aee28f4-06aa-4ede-a75e-d9924b729239-kube-api-access-fw7hl\") pod \"maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:48.745723 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:48.745675    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:18:49.032882 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.032810    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:18:49.130745 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.130717    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:18:49.139772 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.139751    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"]
Jul 31 02:18:49.143327 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.143311    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"
Jul 31 02:18:49.145549 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.145532    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a1d99bab\""
Jul 31 02:18:49.145665 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.145650    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a1d99bab-dockercfg-wslpk\""
Jul 31 02:18:49.154792 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.154773    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"]
Jul 31 02:18:49.161835 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.161818    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7271a83e-c5b7-45cf-ae85-0cbb001855de-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v\" (UID: \"7271a83e-c5b7-45cf-ae85-0cbb001855de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"
Jul 31 02:18:49.161917 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.161854    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp6r4\" (UniqueName: \"kubernetes.io/projected/7271a83e-c5b7-45cf-ae85-0cbb001855de-kube-api-access-zp6r4\") pod \"payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v\" (UID: \"7271a83e-c5b7-45cf-ae85-0cbb001855de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"
Jul 31 02:18:49.161917 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.161883    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aee28f4-06aa-4ede-a75e-d9924b729239-maas-api-tls\") pod \"maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:49.164483 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.164461    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aee28f4-06aa-4ede-a75e-d9924b729239-maas-api-tls\") pod \"maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:49.188682 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.188661    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"]
Jul 31 02:18:49.192084 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.192071    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"
Jul 31 02:18:49.210584 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.210552    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"]
Jul 31 02:18:49.262484 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.262459    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7271a83e-c5b7-45cf-ae85-0cbb001855de-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v\" (UID: \"7271a83e-c5b7-45cf-ae85-0cbb001855de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"
Jul 31 02:18:49.262629 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.262496    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/249d02e7-eb95-431d-82c6-19155ba01515-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z\" (UID: \"249d02e7-eb95-431d-82c6-19155ba01515\") " pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"
Jul 31 02:18:49.262629 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.262521    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp6r4\" (UniqueName: \"kubernetes.io/projected/7271a83e-c5b7-45cf-ae85-0cbb001855de-kube-api-access-zp6r4\") pod \"payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v\" (UID: \"7271a83e-c5b7-45cf-ae85-0cbb001855de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"
Jul 31 02:18:49.262629 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.262544    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzgrp\" (UniqueName: \"kubernetes.io/projected/249d02e7-eb95-431d-82c6-19155ba01515-kube-api-access-gzgrp\") pod \"payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z\" (UID: \"249d02e7-eb95-431d-82c6-19155ba01515\") " pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"
Jul 31 02:18:49.263107 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.263088    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7271a83e-c5b7-45cf-ae85-0cbb001855de-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v\" (UID: \"7271a83e-c5b7-45cf-ae85-0cbb001855de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"
Jul 31 02:18:49.283187 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.283132    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp6r4\" (UniqueName: \"kubernetes.io/projected/7271a83e-c5b7-45cf-ae85-0cbb001855de-kube-api-access-zp6r4\") pod \"payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v\" (UID: \"7271a83e-c5b7-45cf-ae85-0cbb001855de\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"
Jul 31 02:18:49.363340 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.363309    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgrp\" (UniqueName: \"kubernetes.io/projected/249d02e7-eb95-431d-82c6-19155ba01515-kube-api-access-gzgrp\") pod \"payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z\" (UID: \"249d02e7-eb95-431d-82c6-19155ba01515\") " pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"
Jul 31 02:18:49.363445 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.363381    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/249d02e7-eb95-431d-82c6-19155ba01515-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z\" (UID: \"249d02e7-eb95-431d-82c6-19155ba01515\") " pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"
Jul 31 02:18:49.364009 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.363985    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/249d02e7-eb95-431d-82c6-19155ba01515-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z\" (UID: \"249d02e7-eb95-431d-82c6-19155ba01515\") " pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"
Jul 31 02:18:49.372051 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.372030    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgrp\" (UniqueName: \"kubernetes.io/projected/249d02e7-eb95-431d-82c6-19155ba01515-kube-api-access-gzgrp\") pod \"payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z\" (UID: \"249d02e7-eb95-431d-82c6-19155ba01515\") " pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"
Jul 31 02:18:49.409916 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.409893    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:49.454208 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.454184    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"
Jul 31 02:18:49.500652 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.500595    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"
Jul 31 02:18:49.547747 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.547723    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"]
Jul 31 02:18:49.549605 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:18:49.549576    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aee28f4_06aa_4ede_a75e_d9924b729239.slice/crio-2d7f12f7d1997e8e234611f45e1e1790b00e888aaaffffcfc1f329de2ea6d39e WatchSource:0}: Error finding container 2d7f12f7d1997e8e234611f45e1e1790b00e888aaaffffcfc1f329de2ea6d39e: Status 404 returned error can't find the container with id 2d7f12f7d1997e8e234611f45e1e1790b00e888aaaffffcfc1f329de2ea6d39e
Jul 31 02:18:49.551340 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.551272    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 02:18:49.605610 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.605518    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"]
Jul 31 02:18:49.607595 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:18:49.607546    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7271a83e_c5b7_45cf_ae85_0cbb001855de.slice/crio-050fa50fd2b5b09f43c4498cc961aabced2423fd33e1bea787b1cce5d276a7b6 WatchSource:0}: Error finding container 050fa50fd2b5b09f43c4498cc961aabced2423fd33e1bea787b1cce5d276a7b6: Status 404 returned error can't find the container with id 050fa50fd2b5b09f43c4498cc961aabced2423fd33e1bea787b1cce5d276a7b6
Jul 31 02:18:49.653600 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:49.653543    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"]
Jul 31 02:18:49.655302 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:18:49.655276    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249d02e7_eb95_431d_82c6_19155ba01515.slice/crio-8c971edfc0301d1cd803541f0c30f1daeb0fae22668a17d25b4e45c5c106fe44 WatchSource:0}: Error finding container 8c971edfc0301d1cd803541f0c30f1daeb0fae22668a17d25b4e45c5c106fe44: Status 404 returned error can't find the container with id 8c971edfc0301d1cd803541f0c30f1daeb0fae22668a17d25b4e45c5c106fe44
Jul 31 02:18:50.152404 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:50.152369    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z" event={"ID":"249d02e7-eb95-431d-82c6-19155ba01515","Type":"ContainerStarted","Data":"32ffb7bca5b4b8ec83f12c248c9d01aa1514ffcafc2336957e1aa741e163208e"}
Jul 31 02:18:50.152659 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:50.152430    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"
Jul 31 02:18:50.152659 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:50.152453    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z" event={"ID":"249d02e7-eb95-431d-82c6-19155ba01515","Type":"ContainerStarted","Data":"8c971edfc0301d1cd803541f0c30f1daeb0fae22668a17d25b4e45c5c106fe44"}
Jul 31 02:18:50.153795 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:50.153773    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v" event={"ID":"7271a83e-c5b7-45cf-ae85-0cbb001855de","Type":"ContainerStarted","Data":"9623a8c3646b921d7099f23301ece80fa0b8c37914ffa378e9bbcd45db0cdebf"}
Jul 31 02:18:50.153906 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:50.153814    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v" event={"ID":"7271a83e-c5b7-45cf-ae85-0cbb001855de","Type":"ContainerStarted","Data":"050fa50fd2b5b09f43c4498cc961aabced2423fd33e1bea787b1cce5d276a7b6"}
Jul 31 02:18:50.153906 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:50.153840    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"
Jul 31 02:18:50.155057 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:50.155030    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr" event={"ID":"3aee28f4-06aa-4ede-a75e-d9924b729239","Type":"ContainerStarted","Data":"a4c52b679e22ee13ed0220f9092ff473b0268fa070f6cdc0c8a9d1d5b767049b"}
Jul 31 02:18:50.155157 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:50.155068    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr" event={"ID":"3aee28f4-06aa-4ede-a75e-d9924b729239","Type":"ContainerStarted","Data":"2d7f12f7d1997e8e234611f45e1e1790b00e888aaaffffcfc1f329de2ea6d39e"}
Jul 31 02:18:50.155219 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:50.155189    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:18:50.167724 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:50.167668    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z" podStartSLOduration=1.167650518 podStartE2EDuration="1.167650518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:18:50.166360598 +0000 UTC m=+2488.532106226" watchObservedRunningTime="2026-07-31 02:18:50.167650518 +0000 UTC m=+2488.533396144"
Jul 31 02:18:50.184319 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:50.184277    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr" podStartSLOduration=1.6952614750000001 podStartE2EDuration="2.184263322s" totalImagesPullingTime="489.001847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:18:50.182663364 +0000 UTC m=+2488.548408988" watchObservedRunningTime="2026-07-31 02:18:50.184263322 +0000 UTC m=+2488.550008947"
Jul 31 02:18:50.198246 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:50.198206    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v" podStartSLOduration=1.198195081 podStartE2EDuration="1.198195081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:18:50.197449714 +0000 UTC m=+2488.563195343" watchObservedRunningTime="2026-07-31 02:18:50.198195081 +0000 UTC m=+2488.563940706"
Jul 31 02:18:56.164411 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:18:56.164382    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:19:01.161977 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:01.161949    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"
Jul 31 02:19:01.162414 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:01.162113    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"
Jul 31 02:19:12.435725 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:12.435694    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:19:15.715342 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:15.715289    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:19:22.794644 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:22.794616    2556 scope.go:117] "RemoveContainer" containerID="167ed80f138833bfe76012d866a066b899f68e79d7c38122716e72c06b3d6e61"
Jul 31 02:19:41.759451 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:41.759417    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz"]
Jul 31 02:19:41.762550 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:41.762535    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz"
Jul 31 02:19:41.764619 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:41.764597    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:19:41.770015 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:41.769528    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz"]
Jul 31 02:19:41.841377 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:41.841351    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73370bf-53e3-41b0-94f9-834a6cc861f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz\" (UID: \"a73370bf-53e3-41b0-94f9-834a6cc861f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz"
Jul 31 02:19:41.942466 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:41.942448    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73370bf-53e3-41b0-94f9-834a6cc861f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz\" (UID: \"a73370bf-53e3-41b0-94f9-834a6cc861f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz"
Jul 31 02:19:41.943058 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:41.943040    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73370bf-53e3-41b0-94f9-834a6cc861f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz\" (UID: \"a73370bf-53e3-41b0-94f9-834a6cc861f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz"
Jul 31 02:19:42.073487 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:42.073465    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz"
Jul 31 02:19:42.198208 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:42.198184    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz"]
Jul 31 02:19:42.201064 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:19:42.201039    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73370bf_53e3_41b0_94f9_834a6cc861f9.slice/crio-60b1963290848c9e570d165ee35a2e45d318e39db91862a5e6d1656ca4abd5c7 WatchSource:0}: Error finding container 60b1963290848c9e570d165ee35a2e45d318e39db91862a5e6d1656ca4abd5c7: Status 404 returned error can't find the container with id 60b1963290848c9e570d165ee35a2e45d318e39db91862a5e6d1656ca4abd5c7
Jul 31 02:19:42.380613 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:42.380546    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz" event={"ID":"a73370bf-53e3-41b0-94f9-834a6cc861f9","Type":"ContainerStarted","Data":"60b1963290848c9e570d165ee35a2e45d318e39db91862a5e6d1656ca4abd5c7"}
Jul 31 02:19:43.387347 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:43.387310    2556 generic.go:356] "Generic (PLEG): container finished" podID="a73370bf-53e3-41b0-94f9-834a6cc861f9" containerID="4ea6f70ecf7f34804c5f0b061b448f400f3d4a3a9e31a296a9a943a3d198af92" exitCode=0
Jul 31 02:19:43.387703 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:43.387418    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz" event={"ID":"a73370bf-53e3-41b0-94f9-834a6cc861f9","Type":"ContainerDied","Data":"4ea6f70ecf7f34804c5f0b061b448f400f3d4a3a9e31a296a9a943a3d198af92"}
Jul 31 02:19:44.516357 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:44.516332    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz"
Jul 31 02:19:44.663776 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:44.663711    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73370bf-53e3-41b0-94f9-834a6cc861f9-maas-api-service-ca\") pod \"a73370bf-53e3-41b0-94f9-834a6cc861f9\" (UID: \"a73370bf-53e3-41b0-94f9-834a6cc861f9\") "
Jul 31 02:19:44.664046 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:44.664024    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73370bf-53e3-41b0-94f9-834a6cc861f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a73370bf-53e3-41b0-94f9-834a6cc861f9" (UID: "a73370bf-53e3-41b0-94f9-834a6cc861f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:19:44.765044 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:44.765020    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73370bf-53e3-41b0-94f9-834a6cc861f9-maas-api-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:19:45.396284 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:45.396261    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz"
Jul 31 02:19:45.396402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:45.396282    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz" event={"ID":"a73370bf-53e3-41b0-94f9-834a6cc861f9","Type":"ContainerDied","Data":"60b1963290848c9e570d165ee35a2e45d318e39db91862a5e6d1656ca4abd5c7"}
Jul 31 02:19:45.396402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:45.396326    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60b1963290848c9e570d165ee35a2e45d318e39db91862a5e6d1656ca4abd5c7"
Jul 31 02:19:57.130498 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.130464    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"]
Jul 31 02:19:57.130992 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.130726    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr" podUID="3aee28f4-06aa-4ede-a75e-d9924b729239" containerName="maas-api" containerID="cri-o://a4c52b679e22ee13ed0220f9092ff473b0268fa070f6cdc0c8a9d1d5b767049b" gracePeriod=30
Jul 31 02:19:57.156426 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.156403    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:19:57.181287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.181268    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"]
Jul 31 02:19:57.181493 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.181464    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z" podUID="249d02e7-eb95-431d-82c6-19155ba01515" containerName="payload-processing" containerID="cri-o://32ffb7bca5b4b8ec83f12c248c9d01aa1514ffcafc2336957e1aa741e163208e" gracePeriod=30
Jul 31 02:19:57.189032 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.189008    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"]
Jul 31 02:19:57.189212 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.189190    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v" podUID="7271a83e-c5b7-45cf-ae85-0cbb001855de" containerName="payload-pre-processing" containerID="cri-o://9623a8c3646b921d7099f23301ece80fa0b8c37914ffa378e9bbcd45db0cdebf" gracePeriod=30
Jul 31 02:19:57.222089 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.222058    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:19:57.281156 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.281128    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:19:57.319548 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.319509    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz"]
Jul 31 02:19:57.321871 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.321846    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1d99bab-t4rkz"]
Jul 31 02:19:57.346761 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.346724    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:19:57.446917 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.446889    2556 generic.go:356] "Generic (PLEG): container finished" podID="249d02e7-eb95-431d-82c6-19155ba01515" containerID="32ffb7bca5b4b8ec83f12c248c9d01aa1514ffcafc2336957e1aa741e163208e" exitCode=0
Jul 31 02:19:57.447053 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.446962    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z" event={"ID":"249d02e7-eb95-431d-82c6-19155ba01515","Type":"ContainerDied","Data":"32ffb7bca5b4b8ec83f12c248c9d01aa1514ffcafc2336957e1aa741e163208e"}
Jul 31 02:19:57.447053 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.447048    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z" event={"ID":"249d02e7-eb95-431d-82c6-19155ba01515","Type":"ContainerDied","Data":"8c971edfc0301d1cd803541f0c30f1daeb0fae22668a17d25b4e45c5c106fe44"}
Jul 31 02:19:57.447191 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.447073    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c971edfc0301d1cd803541f0c30f1daeb0fae22668a17d25b4e45c5c106fe44"
Jul 31 02:19:57.448935 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.448916    2556 generic.go:356] "Generic (PLEG): container finished" podID="7271a83e-c5b7-45cf-ae85-0cbb001855de" containerID="9623a8c3646b921d7099f23301ece80fa0b8c37914ffa378e9bbcd45db0cdebf" exitCode=0
Jul 31 02:19:57.449054 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.449005    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v" event={"ID":"7271a83e-c5b7-45cf-ae85-0cbb001855de","Type":"ContainerDied","Data":"9623a8c3646b921d7099f23301ece80fa0b8c37914ffa378e9bbcd45db0cdebf"}
Jul 31 02:19:57.450307 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.450289    2556 generic.go:356] "Generic (PLEG): container finished" podID="3aee28f4-06aa-4ede-a75e-d9924b729239" containerID="a4c52b679e22ee13ed0220f9092ff473b0268fa070f6cdc0c8a9d1d5b767049b" exitCode=0
Jul 31 02:19:57.450393 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.450328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr" event={"ID":"3aee28f4-06aa-4ede-a75e-d9924b729239","Type":"ContainerDied","Data":"a4c52b679e22ee13ed0220f9092ff473b0268fa070f6cdc0c8a9d1d5b767049b"}
Jul 31 02:19:57.455664 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.455634    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"
Jul 31 02:19:57.465687 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.465670    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:19:57.478316 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.478301    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"
Jul 31 02:19:57.559079 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.559049    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/249d02e7-eb95-431d-82c6-19155ba01515-plugins-config-volume\") pod \"249d02e7-eb95-431d-82c6-19155ba01515\" (UID: \"249d02e7-eb95-431d-82c6-19155ba01515\") "
Jul 31 02:19:57.559243 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.559091    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aee28f4-06aa-4ede-a75e-d9924b729239-maas-api-tls\") pod \"3aee28f4-06aa-4ede-a75e-d9924b729239\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") "
Jul 31 02:19:57.559243 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.559123    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7271a83e-c5b7-45cf-ae85-0cbb001855de-plugins-config-volume\") pod \"7271a83e-c5b7-45cf-ae85-0cbb001855de\" (UID: \"7271a83e-c5b7-45cf-ae85-0cbb001855de\") "
Jul 31 02:19:57.559243 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.559169    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzgrp\" (UniqueName: \"kubernetes.io/projected/249d02e7-eb95-431d-82c6-19155ba01515-kube-api-access-gzgrp\") pod \"249d02e7-eb95-431d-82c6-19155ba01515\" (UID: \"249d02e7-eb95-431d-82c6-19155ba01515\") "
Jul 31 02:19:57.559243 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.559196    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aee28f4-06aa-4ede-a75e-d9924b729239-openshift-service-ca\") pod \"3aee28f4-06aa-4ede-a75e-d9924b729239\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") "
Jul 31 02:19:57.559243 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.559221    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw7hl\" (UniqueName: \"kubernetes.io/projected/3aee28f4-06aa-4ede-a75e-d9924b729239-kube-api-access-fw7hl\") pod \"3aee28f4-06aa-4ede-a75e-d9924b729239\" (UID: \"3aee28f4-06aa-4ede-a75e-d9924b729239\") "
Jul 31 02:19:57.559243 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.559241    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp6r4\" (UniqueName: \"kubernetes.io/projected/7271a83e-c5b7-45cf-ae85-0cbb001855de-kube-api-access-zp6r4\") pod \"7271a83e-c5b7-45cf-ae85-0cbb001855de\" (UID: \"7271a83e-c5b7-45cf-ae85-0cbb001855de\") "
Jul 31 02:19:57.559533 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.559474    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/249d02e7-eb95-431d-82c6-19155ba01515-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "249d02e7-eb95-431d-82c6-19155ba01515" (UID: "249d02e7-eb95-431d-82c6-19155ba01515"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:19:57.559637 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.559537    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aee28f4-06aa-4ede-a75e-d9924b729239-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3aee28f4-06aa-4ede-a75e-d9924b729239" (UID: "3aee28f4-06aa-4ede-a75e-d9924b729239"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:19:57.559637 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.559595    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7271a83e-c5b7-45cf-ae85-0cbb001855de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7271a83e-c5b7-45cf-ae85-0cbb001855de" (UID: "7271a83e-c5b7-45cf-ae85-0cbb001855de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:19:57.561287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.561258    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249d02e7-eb95-431d-82c6-19155ba01515-kube-api-access-gzgrp" (OuterVolumeSpecName: "kube-api-access-gzgrp") pod "249d02e7-eb95-431d-82c6-19155ba01515" (UID: "249d02e7-eb95-431d-82c6-19155ba01515"). InnerVolumeSpecName "kube-api-access-gzgrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:19:57.561391 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.561284    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aee28f4-06aa-4ede-a75e-d9924b729239-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3aee28f4-06aa-4ede-a75e-d9924b729239" (UID: "3aee28f4-06aa-4ede-a75e-d9924b729239"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:19:57.561391 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.561288    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aee28f4-06aa-4ede-a75e-d9924b729239-kube-api-access-fw7hl" (OuterVolumeSpecName: "kube-api-access-fw7hl") pod "3aee28f4-06aa-4ede-a75e-d9924b729239" (UID: "3aee28f4-06aa-4ede-a75e-d9924b729239"). InnerVolumeSpecName "kube-api-access-fw7hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:19:57.561391 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.561377    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7271a83e-c5b7-45cf-ae85-0cbb001855de-kube-api-access-zp6r4" (OuterVolumeSpecName: "kube-api-access-zp6r4") pod "7271a83e-c5b7-45cf-ae85-0cbb001855de" (UID: "7271a83e-c5b7-45cf-ae85-0cbb001855de"). InnerVolumeSpecName "kube-api-access-zp6r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:19:57.660337 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.660287    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/249d02e7-eb95-431d-82c6-19155ba01515-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:19:57.660337 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.660306    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aee28f4-06aa-4ede-a75e-d9924b729239-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:19:57.660337 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.660315    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7271a83e-c5b7-45cf-ae85-0cbb001855de-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:19:57.660337 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.660322    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzgrp\" (UniqueName: \"kubernetes.io/projected/249d02e7-eb95-431d-82c6-19155ba01515-kube-api-access-gzgrp\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:19:57.660337 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.660330    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aee28f4-06aa-4ede-a75e-d9924b729239-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:19:57.660337 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.660338    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw7hl\" (UniqueName: \"kubernetes.io/projected/3aee28f4-06aa-4ede-a75e-d9924b729239-kube-api-access-fw7hl\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:19:57.660577 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:57.660346    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp6r4\" (UniqueName: \"kubernetes.io/projected/7271a83e-c5b7-45cf-ae85-0cbb001855de-kube-api-access-zp6r4\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:19:58.151087 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.151063    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a73370bf-53e3-41b0-94f9-834a6cc861f9" path="/var/lib/kubelet/pods/a73370bf-53e3-41b0-94f9-834a6cc861f9/volumes"
Jul 31 02:19:58.456440 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.456382    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"
Jul 31 02:19:58.456440 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.456387    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v" event={"ID":"7271a83e-c5b7-45cf-ae85-0cbb001855de","Type":"ContainerDied","Data":"050fa50fd2b5b09f43c4498cc961aabced2423fd33e1bea787b1cce5d276a7b6"}
Jul 31 02:19:58.456640 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.456487    2556 scope.go:117] "RemoveContainer" containerID="9623a8c3646b921d7099f23301ece80fa0b8c37914ffa378e9bbcd45db0cdebf"
Jul 31 02:19:58.457771 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.457754    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"
Jul 31 02:19:58.457900 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.457747    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr" event={"ID":"3aee28f4-06aa-4ede-a75e-d9924b729239","Type":"ContainerDied","Data":"2d7f12f7d1997e8e234611f45e1e1790b00e888aaaffffcfc1f329de2ea6d39e"}
Jul 31 02:19:58.458154 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.458088    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"
Jul 31 02:19:58.464880 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.464865    2556 scope.go:117] "RemoveContainer" containerID="a4c52b679e22ee13ed0220f9092ff473b0268fa070f6cdc0c8a9d1d5b767049b"
Jul 31 02:19:58.475851 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.475831    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"]
Jul 31 02:19:58.479163 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.479142    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1d99bab-6b749fb8b8-cgg8v"]
Jul 31 02:19:58.487967 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.487949    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"]
Jul 31 02:19:58.491886 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.491868    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1d99bab-5c6f6bc595-rbdqr"]
Jul 31 02:19:58.501134 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.501113    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"]
Jul 31 02:19:58.504802 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:19:58.504784    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1d99bab-64674cb798-sg27z"]
Jul 31 02:20:00.150167 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:20:00.150141    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="249d02e7-eb95-431d-82c6-19155ba01515" path="/var/lib/kubelet/pods/249d02e7-eb95-431d-82c6-19155ba01515/volumes"
Jul 31 02:20:00.150489 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:20:00.150475    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aee28f4-06aa-4ede-a75e-d9924b729239" path="/var/lib/kubelet/pods/3aee28f4-06aa-4ede-a75e-d9924b729239/volumes"
Jul 31 02:20:00.150775 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:20:00.150764    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7271a83e-c5b7-45cf-ae85-0cbb001855de" path="/var/lib/kubelet/pods/7271a83e-c5b7-45cf-ae85-0cbb001855de/volumes"
Jul 31 02:20:00.769204 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:20:00.769170    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:20:05.783795 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:20:05.783767    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:20:22.810502 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:20:22.810475    2556 scope.go:117] "RemoveContainer" containerID="49c1267d00b86415f650acc258f42d714899201623ec3cd52be2fc52a74da8b4"
Jul 31 02:20:42.007471 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:20:42.007433    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:20:43.189254 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:20:43.189175    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:20:48.182449 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:20:48.182408    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:20:48.283181 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:20:48.283146    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:20:48.455720 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:20:48.455655    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:20:48.698320 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:20:48.698295    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:00.006524 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:00.006492    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:00.104741 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:00.104712    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:00.206010 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:00.205981    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:00.804546 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:00.804512    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:00.899156 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:00.899132    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:01.000434 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:01.000413    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:02.233375 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:02.233341    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:08.655927 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:08.655894    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:08.800309 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:08.800275    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:09.326351 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:09.326325    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:16.105843 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:16.105807    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:16.215265 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:16.215239    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:16.328593 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:16.328553    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:26.847862 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:26.847829    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:27.082133 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:27.082091    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:48.070656 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:48.070629    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:53.953703 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:53.953669    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:54.053929 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:54.053900    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:54.333516 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:54.333493    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:54.669922 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:54.669852    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:54.928854 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:54.928792    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:57.459082 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:57.459053    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:57.534373 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:57.534329    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:58.635424 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.635384    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"]
Jul 31 02:21:58.635890 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.635875    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73370bf-53e3-41b0-94f9-834a6cc861f9" containerName="revoke-keys"
Jul 31 02:21:58.635954 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.635895    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73370bf-53e3-41b0-94f9-834a6cc861f9" containerName="revoke-keys"
Jul 31 02:21:58.635954 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.635916    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="249d02e7-eb95-431d-82c6-19155ba01515" containerName="payload-processing"
Jul 31 02:21:58.635954 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.635925    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="249d02e7-eb95-431d-82c6-19155ba01515" containerName="payload-processing"
Jul 31 02:21:58.635954 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.635941    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7271a83e-c5b7-45cf-ae85-0cbb001855de" containerName="payload-pre-processing"
Jul 31 02:21:58.635954 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.635951    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7271a83e-c5b7-45cf-ae85-0cbb001855de" containerName="payload-pre-processing"
Jul 31 02:21:58.636206 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.635969    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aee28f4-06aa-4ede-a75e-d9924b729239" containerName="maas-api"
Jul 31 02:21:58.636206 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.635977    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aee28f4-06aa-4ede-a75e-d9924b729239" containerName="maas-api"
Jul 31 02:21:58.636206 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.636046    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a73370bf-53e3-41b0-94f9-834a6cc861f9" containerName="revoke-keys"
Jul 31 02:21:58.636206 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.636062    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7271a83e-c5b7-45cf-ae85-0cbb001855de" containerName="payload-pre-processing"
Jul 31 02:21:58.636206 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.636071    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="249d02e7-eb95-431d-82c6-19155ba01515" containerName="payload-processing"
Jul 31 02:21:58.636206 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.636080    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aee28f4-06aa-4ede-a75e-d9924b729239" containerName="maas-api"
Jul 31 02:21:58.639228 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.639209    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:21:58.641339 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.641322    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8af864-serving-cert\""
Jul 31 02:21:58.649080 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.649037    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"]
Jul 31 02:21:58.669877 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.669860    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:58.798723 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.798701    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a232efe8-1ff7-4370-8129-58adf4438cd8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s\" (UID: \"a232efe8-1ff7-4370-8129-58adf4438cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:21:58.798835 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.798726    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs9c6\" (UniqueName: \"kubernetes.io/projected/a232efe8-1ff7-4370-8129-58adf4438cd8-kube-api-access-qs9c6\") pod \"maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s\" (UID: \"a232efe8-1ff7-4370-8129-58adf4438cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:21:58.798835 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.798752    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a232efe8-1ff7-4370-8129-58adf4438cd8-maas-api-tls\") pod \"maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s\" (UID: \"a232efe8-1ff7-4370-8129-58adf4438cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:21:58.898331 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.898277    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:58.899461 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.899429    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a232efe8-1ff7-4370-8129-58adf4438cd8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s\" (UID: \"a232efe8-1ff7-4370-8129-58adf4438cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:21:58.899602 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.899465    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9c6\" (UniqueName: \"kubernetes.io/projected/a232efe8-1ff7-4370-8129-58adf4438cd8-kube-api-access-qs9c6\") pod \"maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s\" (UID: \"a232efe8-1ff7-4370-8129-58adf4438cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:21:58.899602 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.899506    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a232efe8-1ff7-4370-8129-58adf4438cd8-maas-api-tls\") pod \"maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s\" (UID: \"a232efe8-1ff7-4370-8129-58adf4438cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:21:58.900180 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.900075    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a232efe8-1ff7-4370-8129-58adf4438cd8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s\" (UID: \"a232efe8-1ff7-4370-8129-58adf4438cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:21:58.901781 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.901763    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a232efe8-1ff7-4370-8129-58adf4438cd8-maas-api-tls\") pod \"maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s\" (UID: \"a232efe8-1ff7-4370-8129-58adf4438cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:21:58.907210 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.907191    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9c6\" (UniqueName: \"kubernetes.io/projected/a232efe8-1ff7-4370-8129-58adf4438cd8-kube-api-access-qs9c6\") pod \"maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s\" (UID: \"a232efe8-1ff7-4370-8129-58adf4438cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:21:58.950394 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:58.950375    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:21:59.076166 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.076146    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"]
Jul 31 02:21:59.078388 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:21:59.078361    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda232efe8_1ff7_4370_8129_58adf4438cd8.slice/crio-e4a48353e3043a3abf58be7d738471d378a42bde90d72bd0398f7811302f4fda WatchSource:0}: Error finding container e4a48353e3043a3abf58be7d738471d378a42bde90d72bd0398f7811302f4fda: Status 404 returned error can't find the container with id e4a48353e3043a3abf58be7d738471d378a42bde90d72bd0398f7811302f4fda
Jul 31 02:21:59.121269 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.121246    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:59.219443 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.219375    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:21:59.236428 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.236404    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"]
Jul 31 02:21:59.240769 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.240754    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"
Jul 31 02:21:59.243101 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.243079    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8af864\""
Jul 31 02:21:59.243183 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.243118    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8af864-dockercfg-9slrs\""
Jul 31 02:21:59.252431 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.252408    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"]
Jul 31 02:21:59.266203 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.266167    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"]
Jul 31 02:21:59.269539 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.269519    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"
Jul 31 02:21:59.278014 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.277995    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"]
Jul 31 02:21:59.404769 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.404743    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lcng\" (UniqueName: \"kubernetes.io/projected/076c4581-b7a7-4b55-b2d7-e86be2214cba-kube-api-access-9lcng\") pod \"payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58\" (UID: \"076c4581-b7a7-4b55-b2d7-e86be2214cba\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"
Jul 31 02:21:59.404917 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.404805    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzpsp\" (UniqueName: \"kubernetes.io/projected/9f5884d2-6675-40bb-aab0-e9eb15604d96-kube-api-access-pzpsp\") pod \"payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh\" (UID: \"9f5884d2-6675-40bb-aab0-e9eb15604d96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"
Jul 31 02:21:59.404917 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.404848    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076c4581-b7a7-4b55-b2d7-e86be2214cba-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58\" (UID: \"076c4581-b7a7-4b55-b2d7-e86be2214cba\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"
Jul 31 02:21:59.404917 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.404873    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5884d2-6675-40bb-aab0-e9eb15604d96-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh\" (UID: \"9f5884d2-6675-40bb-aab0-e9eb15604d96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"
Jul 31 02:21:59.505947 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.505878    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5884d2-6675-40bb-aab0-e9eb15604d96-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh\" (UID: \"9f5884d2-6675-40bb-aab0-e9eb15604d96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"
Jul 31 02:21:59.505947 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.505932    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lcng\" (UniqueName: \"kubernetes.io/projected/076c4581-b7a7-4b55-b2d7-e86be2214cba-kube-api-access-9lcng\") pod \"payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58\" (UID: \"076c4581-b7a7-4b55-b2d7-e86be2214cba\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"
Jul 31 02:21:59.506078 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.506012    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzpsp\" (UniqueName: \"kubernetes.io/projected/9f5884d2-6675-40bb-aab0-e9eb15604d96-kube-api-access-pzpsp\") pod \"payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh\" (UID: \"9f5884d2-6675-40bb-aab0-e9eb15604d96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"
Jul 31 02:21:59.506078 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.506050    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076c4581-b7a7-4b55-b2d7-e86be2214cba-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58\" (UID: \"076c4581-b7a7-4b55-b2d7-e86be2214cba\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"
Jul 31 02:21:59.506630 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.506611    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5884d2-6675-40bb-aab0-e9eb15604d96-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh\" (UID: \"9f5884d2-6675-40bb-aab0-e9eb15604d96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"
Jul 31 02:21:59.506718 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.506700    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076c4581-b7a7-4b55-b2d7-e86be2214cba-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58\" (UID: \"076c4581-b7a7-4b55-b2d7-e86be2214cba\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"
Jul 31 02:21:59.513483 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.513460    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzpsp\" (UniqueName: \"kubernetes.io/projected/9f5884d2-6675-40bb-aab0-e9eb15604d96-kube-api-access-pzpsp\") pod \"payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh\" (UID: \"9f5884d2-6675-40bb-aab0-e9eb15604d96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"
Jul 31 02:21:59.513615 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.513595    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lcng\" (UniqueName: \"kubernetes.io/projected/076c4581-b7a7-4b55-b2d7-e86be2214cba-kube-api-access-9lcng\") pod \"payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58\" (UID: \"076c4581-b7a7-4b55-b2d7-e86be2214cba\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"
Jul 31 02:21:59.551320 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.551286    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"
Jul 31 02:21:59.578806 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.578786    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"
Jul 31 02:21:59.722607 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.722583    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"]
Jul 31 02:21:59.724164 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:21:59.724129    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076c4581_b7a7_4b55_b2d7_e86be2214cba.slice/crio-561c58ccd5f65790e011a79501bdcd4531dfdbd8e7fd195ab471a1dcc7c591f8 WatchSource:0}: Error finding container 561c58ccd5f65790e011a79501bdcd4531dfdbd8e7fd195ab471a1dcc7c591f8: Status 404 returned error can't find the container with id 561c58ccd5f65790e011a79501bdcd4531dfdbd8e7fd195ab471a1dcc7c591f8
Jul 31 02:21:59.900401 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.900316    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"]
Jul 31 02:21:59.902894 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:21:59.902855    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5884d2_6675_40bb_aab0_e9eb15604d96.slice/crio-ace1b03ffc42a8fdb15850d45e6cbcf7de82c6751b15a943a17750c617589536 WatchSource:0}: Error finding container ace1b03ffc42a8fdb15850d45e6cbcf7de82c6751b15a943a17750c617589536: Status 404 returned error can't find the container with id ace1b03ffc42a8fdb15850d45e6cbcf7de82c6751b15a943a17750c617589536
Jul 31 02:21:59.949407 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.949334    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh" event={"ID":"9f5884d2-6675-40bb-aab0-e9eb15604d96","Type":"ContainerStarted","Data":"ace1b03ffc42a8fdb15850d45e6cbcf7de82c6751b15a943a17750c617589536"}
Jul 31 02:21:59.951150 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.951124    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s" event={"ID":"a232efe8-1ff7-4370-8129-58adf4438cd8","Type":"ContainerStarted","Data":"95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343"}
Jul 31 02:21:59.951278 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.951261    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:21:59.951350 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.951317    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s" event={"ID":"a232efe8-1ff7-4370-8129-58adf4438cd8","Type":"ContainerStarted","Data":"e4a48353e3043a3abf58be7d738471d378a42bde90d72bd0398f7811302f4fda"}
Jul 31 02:21:59.952686 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.952638    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58" event={"ID":"076c4581-b7a7-4b55-b2d7-e86be2214cba","Type":"ContainerStarted","Data":"f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6"}
Jul 31 02:21:59.952773 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.952711    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58" event={"ID":"076c4581-b7a7-4b55-b2d7-e86be2214cba","Type":"ContainerStarted","Data":"561c58ccd5f65790e011a79501bdcd4531dfdbd8e7fd195ab471a1dcc7c591f8"}
Jul 31 02:21:59.952773 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.952741    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"
Jul 31 02:21:59.966708 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.966668    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s" podStartSLOduration=1.510237608 podStartE2EDuration="1.966651076s" totalImagesPullingTime="456.413468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:21:59.965182768 +0000 UTC m=+2678.330928394" watchObservedRunningTime="2026-07-31 02:21:59.966651076 +0000 UTC m=+2678.332396702"
Jul 31 02:21:59.979715 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:21:59.979677    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58" podStartSLOduration=0.979663159 podStartE2EDuration="979.663159ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:21:59.978762021 +0000 UTC m=+2678.344507648" watchObservedRunningTime="2026-07-31 02:21:59.979663159 +0000 UTC m=+2678.345408785"
Jul 31 02:22:00.958850 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:00.958810    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh" event={"ID":"9f5884d2-6675-40bb-aab0-e9eb15604d96","Type":"ContainerStarted","Data":"73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c"}
Jul 31 02:22:00.959241 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:00.958906    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"
Jul 31 02:22:00.974088 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:00.974000    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh" podStartSLOduration=1.973987299 podStartE2EDuration="1.973987299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:22:00.973322837 +0000 UTC m=+2679.339068464" watchObservedRunningTime="2026-07-31 02:22:00.973987299 +0000 UTC m=+2679.339732924"
Jul 31 02:22:05.964337 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:05.964312    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:22:10.960715 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:10.960682    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"
Jul 31 02:22:11.964851 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:11.964825    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"
Jul 31 02:22:22.489044 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:22.489016    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:22:22.833925 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:22.833903    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:22:22.839575 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:22.839542    2556 scope.go:117] "RemoveContainer" containerID="c24ceeb4c9f9b174b64dba7366db943abfd89467387b9a5023dee88f0754ea13"
Jul 31 02:22:23.145988 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:23.145914    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:22:23.242337 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:23.242311    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:22:25.600614 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:25.600582    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:22:25.703411 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:25.703377    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:22:26.675482 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.675451    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"]
Jul 31 02:22:26.680243 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.680223    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:26.682385 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.682360    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f88d9e-serving-cert\""
Jul 31 02:22:26.688284 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.688260    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"]
Jul 31 02:22:26.701450 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.701422    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78130c40-1619-43ec-b442-745adc9266f1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:26.701576 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.701498    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cnhz\" (UniqueName: \"kubernetes.io/projected/78130c40-1619-43ec-b442-745adc9266f1-kube-api-access-7cnhz\") pod \"maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:26.701576 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.701531    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78130c40-1619-43ec-b442-745adc9266f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:26.712507 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.712482    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:22:26.802036 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.802009    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cnhz\" (UniqueName: \"kubernetes.io/projected/78130c40-1619-43ec-b442-745adc9266f1-kube-api-access-7cnhz\") pod \"maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:26.802150 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.802040    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78130c40-1619-43ec-b442-745adc9266f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:26.802150 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.802075    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78130c40-1619-43ec-b442-745adc9266f1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:26.802233 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:22:26.802156    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-serving-cert: secret "maas-api-e2e-inf-b-f88d9e-serving-cert" not found
Jul 31 02:22:26.802233 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:22:26.802219    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78130c40-1619-43ec-b442-745adc9266f1-maas-api-tls podName:78130c40-1619-43ec-b442-745adc9266f1 nodeName:}" failed. No retries permitted until 2026-07-31 02:22:27.302200691 +0000 UTC m=+2705.667946300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78130c40-1619-43ec-b442-745adc9266f1-maas-api-tls") pod "maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg" (UID: "78130c40-1619-43ec-b442-745adc9266f1") : secret "maas-api-e2e-inf-b-f88d9e-serving-cert" not found
Jul 31 02:22:26.802617 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.802600    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78130c40-1619-43ec-b442-745adc9266f1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:26.812652 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:26.812631    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cnhz\" (UniqueName: \"kubernetes.io/projected/78130c40-1619-43ec-b442-745adc9266f1-kube-api-access-7cnhz\") pod \"maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:27.293616 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.293580    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:22:27.305786 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.305762    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78130c40-1619-43ec-b442-745adc9266f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:27.305904 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:22:27.305890    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-serving-cert: secret "maas-api-e2e-inf-b-f88d9e-serving-cert" not found
Jul 31 02:22:27.305952 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:22:27.305946    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78130c40-1619-43ec-b442-745adc9266f1-maas-api-tls podName:78130c40-1619-43ec-b442-745adc9266f1 nodeName:}" failed. No retries permitted until 2026-07-31 02:22:28.305929029 +0000 UTC m=+2706.671674649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78130c40-1619-43ec-b442-745adc9266f1-maas-api-tls") pod "maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg" (UID: "78130c40-1619-43ec-b442-745adc9266f1") : secret "maas-api-e2e-inf-b-f88d9e-serving-cert" not found
Jul 31 02:22:27.393983 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.393956    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:22:27.400867 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.400844    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"]
Jul 31 02:22:27.404223 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.404208    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"
Jul 31 02:22:27.406658 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.406635    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f88d9e-dockercfg-p8xs7\""
Jul 31 02:22:27.406766 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.406658    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f88d9e\""
Jul 31 02:22:27.416155 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.416116    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"]
Jul 31 02:22:27.445583 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.445532    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"]
Jul 31 02:22:27.448698 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.448680    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"
Jul 31 02:22:27.459001 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.458976    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"]
Jul 31 02:22:27.461175 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.461156    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:22:27.507110 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.507085    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z59vr\" (UniqueName: \"kubernetes.io/projected/406f0c46-2c63-460a-9e65-923b450d2a57-kube-api-access-z59vr\") pod \"payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm\" (UID: \"406f0c46-2c63-460a-9e65-923b450d2a57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"
Jul 31 02:22:27.507235 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.507140    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55757cb-223a-44f5-bf06-bb61f945c426-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd\" (UID: \"d55757cb-223a-44f5-bf06-bb61f945c426\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"
Jul 31 02:22:27.507235 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.507210    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wdtw\" (UniqueName: \"kubernetes.io/projected/d55757cb-223a-44f5-bf06-bb61f945c426-kube-api-access-8wdtw\") pod \"payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd\" (UID: \"d55757cb-223a-44f5-bf06-bb61f945c426\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"
Jul 31 02:22:27.507319 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.507256    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406f0c46-2c63-460a-9e65-923b450d2a57-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm\" (UID: \"406f0c46-2c63-460a-9e65-923b450d2a57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"
Jul 31 02:22:27.608036 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.608010    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55757cb-223a-44f5-bf06-bb61f945c426-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd\" (UID: \"d55757cb-223a-44f5-bf06-bb61f945c426\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"
Jul 31 02:22:27.608164 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.608044    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wdtw\" (UniqueName: \"kubernetes.io/projected/d55757cb-223a-44f5-bf06-bb61f945c426-kube-api-access-8wdtw\") pod \"payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd\" (UID: \"d55757cb-223a-44f5-bf06-bb61f945c426\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"
Jul 31 02:22:27.608164 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.608066    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406f0c46-2c63-460a-9e65-923b450d2a57-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm\" (UID: \"406f0c46-2c63-460a-9e65-923b450d2a57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"
Jul 31 02:22:27.608164 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.608131    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z59vr\" (UniqueName: \"kubernetes.io/projected/406f0c46-2c63-460a-9e65-923b450d2a57-kube-api-access-z59vr\") pod \"payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm\" (UID: \"406f0c46-2c63-460a-9e65-923b450d2a57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"
Jul 31 02:22:27.608832 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.608812    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406f0c46-2c63-460a-9e65-923b450d2a57-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm\" (UID: \"406f0c46-2c63-460a-9e65-923b450d2a57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"
Jul 31 02:22:27.608921 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.608835    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55757cb-223a-44f5-bf06-bb61f945c426-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd\" (UID: \"d55757cb-223a-44f5-bf06-bb61f945c426\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"
Jul 31 02:22:27.616372 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.616346    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wdtw\" (UniqueName: \"kubernetes.io/projected/d55757cb-223a-44f5-bf06-bb61f945c426-kube-api-access-8wdtw\") pod \"payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd\" (UID: \"d55757cb-223a-44f5-bf06-bb61f945c426\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"
Jul 31 02:22:27.616470 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.616450    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z59vr\" (UniqueName: \"kubernetes.io/projected/406f0c46-2c63-460a-9e65-923b450d2a57-kube-api-access-z59vr\") pod \"payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm\" (UID: \"406f0c46-2c63-460a-9e65-923b450d2a57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"
Jul 31 02:22:27.714928 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.714908    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"
Jul 31 02:22:27.758104 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.758074    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"
Jul 31 02:22:27.854897 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.854869    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"]
Jul 31 02:22:27.903215 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:27.903188    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"]
Jul 31 02:22:27.909647 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:22:27.909609    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406f0c46_2c63_460a_9e65_923b450d2a57.slice/crio-fef7d9bd7f201a5959bc63e8eda7f83892cc457215a4096c5d9c7e7095ff4b07 WatchSource:0}: Error finding container fef7d9bd7f201a5959bc63e8eda7f83892cc457215a4096c5d9c7e7095ff4b07: Status 404 returned error can't find the container with id fef7d9bd7f201a5959bc63e8eda7f83892cc457215a4096c5d9c7e7095ff4b07
Jul 31 02:22:28.088278 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:28.088219    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd" event={"ID":"d55757cb-223a-44f5-bf06-bb61f945c426","Type":"ContainerStarted","Data":"93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752"}
Jul 31 02:22:28.088452 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:28.088295    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd" event={"ID":"d55757cb-223a-44f5-bf06-bb61f945c426","Type":"ContainerStarted","Data":"bf74b9e91883a555da03680e008e493ad9c59aeb7c19d02dc314cdbd3351bfac"}
Jul 31 02:22:28.088452 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:28.088341    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"
Jul 31 02:22:28.090235 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:28.090217    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm" event={"ID":"406f0c46-2c63-460a-9e65-923b450d2a57","Type":"ContainerStarted","Data":"68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6"}
Jul 31 02:22:28.090377 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:28.090297    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm" event={"ID":"406f0c46-2c63-460a-9e65-923b450d2a57","Type":"ContainerStarted","Data":"fef7d9bd7f201a5959bc63e8eda7f83892cc457215a4096c5d9c7e7095ff4b07"}
Jul 31 02:22:28.090377 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:28.090350    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"
Jul 31 02:22:28.105586 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:28.105525    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd" podStartSLOduration=1.105513339 podStartE2EDuration="1.105513339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:22:28.103977421 +0000 UTC m=+2706.469723048" watchObservedRunningTime="2026-07-31 02:22:28.105513339 +0000 UTC m=+2706.471258995"
Jul 31 02:22:28.119362 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:28.119224    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm" podStartSLOduration=1.119206575 podStartE2EDuration="1.119206575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:22:28.118970915 +0000 UTC m=+2706.484716541" watchObservedRunningTime="2026-07-31 02:22:28.119206575 +0000 UTC m=+2706.484952202"
Jul 31 02:22:28.314641 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:28.314614    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78130c40-1619-43ec-b442-745adc9266f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:28.316937 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:28.316914    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78130c40-1619-43ec-b442-745adc9266f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:28.491618 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:28.491521    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:28.625508 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:28.625483    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"]
Jul 31 02:22:28.627580 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:22:28.627535    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78130c40_1619_43ec_b442_745adc9266f1.slice/crio-3e988b6d87aa47f4f808dbcd434f4560ba5955034b1312d58518585ade724975 WatchSource:0}: Error finding container 3e988b6d87aa47f4f808dbcd434f4560ba5955034b1312d58518585ade724975: Status 404 returned error can't find the container with id 3e988b6d87aa47f4f808dbcd434f4560ba5955034b1312d58518585ade724975
Jul 31 02:22:29.095780 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:29.095743    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg" event={"ID":"78130c40-1619-43ec-b442-745adc9266f1","Type":"ContainerStarted","Data":"c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6"}
Jul 31 02:22:29.096137 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:29.095806    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg" event={"ID":"78130c40-1619-43ec-b442-745adc9266f1","Type":"ContainerStarted","Data":"3e988b6d87aa47f4f808dbcd434f4560ba5955034b1312d58518585ade724975"}
Jul 31 02:22:29.096187 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:29.096142    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:29.112087 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:29.112044    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg" podStartSLOduration=2.779659606 podStartE2EDuration="3.112030922s" totalImagesPullingTime="332.371316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:22:29.11106208 +0000 UTC m=+2707.476807706" watchObservedRunningTime="2026-07-31 02:22:29.112030922 +0000 UTC m=+2707.477776548"
Jul 31 02:22:36.109767 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:36.109741    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:22:39.097720 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:39.097689    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"
Jul 31 02:22:39.098052 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:39.097752    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"
Jul 31 02:22:42.856990 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:42.856961    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:22:50.614768 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:22:50.614730    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:23:00.313745 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:23:00.313696    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:23:22.595751 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:23:22.595718    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:23:31.670825 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:23:31.670776    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:23:38.565798 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:23:38.565744    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:23:38.617431 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:23:38.617381    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:05.295206 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:05.295175    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:06.181284 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:06.181253    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:06.302347 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:06.302309    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99"]
Jul 31 02:24:06.305872 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:06.305849    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99"
Jul 31 02:24:06.307913 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:06.307896    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:24:06.311877 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:06.311706    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99"]
Jul 31 02:24:06.412545 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:06.412520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4330814c-58ee-44ff-befb-7ee65d481e1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8af864-zvq99\" (UID: \"4330814c-58ee-44ff-befb-7ee65d481e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99"
Jul 31 02:24:06.513401 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:06.513344    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4330814c-58ee-44ff-befb-7ee65d481e1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8af864-zvq99\" (UID: \"4330814c-58ee-44ff-befb-7ee65d481e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99"
Jul 31 02:24:06.513905 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:06.513886    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4330814c-58ee-44ff-befb-7ee65d481e1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8af864-zvq99\" (UID: \"4330814c-58ee-44ff-befb-7ee65d481e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99"
Jul 31 02:24:06.618513 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:06.618485    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99"
Jul 31 02:24:06.745770 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:06.745747    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99"]
Jul 31 02:24:06.747712 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:24:06.747675    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4330814c_58ee_44ff_befb_7ee65d481e1c.slice/crio-6c8fecd0503670ebedc403bb8e0291dc97d13f61ad0ee8dec44da4cbb8859a5b WatchSource:0}: Error finding container 6c8fecd0503670ebedc403bb8e0291dc97d13f61ad0ee8dec44da4cbb8859a5b: Status 404 returned error can't find the container with id 6c8fecd0503670ebedc403bb8e0291dc97d13f61ad0ee8dec44da4cbb8859a5b
Jul 31 02:24:07.503993 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:07.503963    2556 generic.go:356] "Generic (PLEG): container finished" podID="4330814c-58ee-44ff-befb-7ee65d481e1c" containerID="5587ce914b516a1a71173b4fb88dc83aea0428bca02c2b6df794520266b52386" exitCode=0
Jul 31 02:24:07.504320 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:07.504049    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99" event={"ID":"4330814c-58ee-44ff-befb-7ee65d481e1c","Type":"ContainerDied","Data":"5587ce914b516a1a71173b4fb88dc83aea0428bca02c2b6df794520266b52386"}
Jul 31 02:24:07.504320 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:07.504094    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99" event={"ID":"4330814c-58ee-44ff-befb-7ee65d481e1c","Type":"ContainerStarted","Data":"6c8fecd0503670ebedc403bb8e0291dc97d13f61ad0ee8dec44da4cbb8859a5b"}
Jul 31 02:24:08.625239 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:08.625218    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99"
Jul 31 02:24:08.730498 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:08.730473    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4330814c-58ee-44ff-befb-7ee65d481e1c-maas-api-service-ca\") pod \"4330814c-58ee-44ff-befb-7ee65d481e1c\" (UID: \"4330814c-58ee-44ff-befb-7ee65d481e1c\") "
Jul 31 02:24:08.730784 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:08.730764    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4330814c-58ee-44ff-befb-7ee65d481e1c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4330814c-58ee-44ff-befb-7ee65d481e1c" (UID: "4330814c-58ee-44ff-befb-7ee65d481e1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:24:08.831515 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:08.831496    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4330814c-58ee-44ff-befb-7ee65d481e1c-maas-api-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:09.513842 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:09.513807    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99" event={"ID":"4330814c-58ee-44ff-befb-7ee65d481e1c","Type":"ContainerDied","Data":"6c8fecd0503670ebedc403bb8e0291dc97d13f61ad0ee8dec44da4cbb8859a5b"}
Jul 31 02:24:09.513986 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:09.513876    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c8fecd0503670ebedc403bb8e0291dc97d13f61ad0ee8dec44da4cbb8859a5b"
Jul 31 02:24:09.513986 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:09.513820    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99"
Jul 31 02:24:16.655959 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:16.655924    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"]
Jul 31 02:24:16.656422 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:16.656165    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s" podUID="a232efe8-1ff7-4370-8129-58adf4438cd8" containerName="maas-api" containerID="cri-o://95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343" gracePeriod=30
Jul 31 02:24:16.687696 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:16.687669    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:16.709723 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:16.709697    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"]
Jul 31 02:24:16.709944 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:16.709903    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58" podUID="076c4581-b7a7-4b55-b2d7-e86be2214cba" containerName="payload-processing" containerID="cri-o://f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6" gracePeriod=30
Jul 31 02:24:16.715951 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:16.715928    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"]
Jul 31 02:24:16.716183 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:16.716158    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh" podUID="9f5884d2-6675-40bb-aab0-e9eb15604d96" containerName="payload-pre-processing" containerID="cri-o://73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c" gracePeriod=30
Jul 31 02:24:16.788267 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:16.788244    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:16.854436 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:16.854369    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99"]
Jul 31 02:24:16.856710 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:16.856683    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8af864-zvq99"]
Jul 31 02:24:16.857511 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:16.857489    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:17.017112 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.017092    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"
Jul 31 02:24:17.020148 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.020130    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"
Jul 31 02:24:17.089624 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.089550    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzpsp\" (UniqueName: \"kubernetes.io/projected/9f5884d2-6675-40bb-aab0-e9eb15604d96-kube-api-access-pzpsp\") pod \"9f5884d2-6675-40bb-aab0-e9eb15604d96\" (UID: \"9f5884d2-6675-40bb-aab0-e9eb15604d96\") "
Jul 31 02:24:17.089803 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.089665    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076c4581-b7a7-4b55-b2d7-e86be2214cba-plugins-config-volume\") pod \"076c4581-b7a7-4b55-b2d7-e86be2214cba\" (UID: \"076c4581-b7a7-4b55-b2d7-e86be2214cba\") "
Jul 31 02:24:17.089803 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.089690    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5884d2-6675-40bb-aab0-e9eb15604d96-plugins-config-volume\") pod \"9f5884d2-6675-40bb-aab0-e9eb15604d96\" (UID: \"9f5884d2-6675-40bb-aab0-e9eb15604d96\") "
Jul 31 02:24:17.089917 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.089805    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lcng\" (UniqueName: \"kubernetes.io/projected/076c4581-b7a7-4b55-b2d7-e86be2214cba-kube-api-access-9lcng\") pod \"076c4581-b7a7-4b55-b2d7-e86be2214cba\" (UID: \"076c4581-b7a7-4b55-b2d7-e86be2214cba\") "
Jul 31 02:24:17.090153 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.090112    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076c4581-b7a7-4b55-b2d7-e86be2214cba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "076c4581-b7a7-4b55-b2d7-e86be2214cba" (UID: "076c4581-b7a7-4b55-b2d7-e86be2214cba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:24:17.090314 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.090160    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5884d2-6675-40bb-aab0-e9eb15604d96-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f5884d2-6675-40bb-aab0-e9eb15604d96" (UID: "9f5884d2-6675-40bb-aab0-e9eb15604d96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:24:17.092062 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.092035    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5884d2-6675-40bb-aab0-e9eb15604d96-kube-api-access-pzpsp" (OuterVolumeSpecName: "kube-api-access-pzpsp") pod "9f5884d2-6675-40bb-aab0-e9eb15604d96" (UID: "9f5884d2-6675-40bb-aab0-e9eb15604d96"). InnerVolumeSpecName "kube-api-access-pzpsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:24:17.092508 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.092487    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076c4581-b7a7-4b55-b2d7-e86be2214cba-kube-api-access-9lcng" (OuterVolumeSpecName: "kube-api-access-9lcng") pod "076c4581-b7a7-4b55-b2d7-e86be2214cba" (UID: "076c4581-b7a7-4b55-b2d7-e86be2214cba"). InnerVolumeSpecName "kube-api-access-9lcng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:24:17.111501 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.111479    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:24:17.191124 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.191077    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a232efe8-1ff7-4370-8129-58adf4438cd8-maas-api-tls\") pod \"a232efe8-1ff7-4370-8129-58adf4438cd8\" (UID: \"a232efe8-1ff7-4370-8129-58adf4438cd8\") "
Jul 31 02:24:17.191124 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.191106    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs9c6\" (UniqueName: \"kubernetes.io/projected/a232efe8-1ff7-4370-8129-58adf4438cd8-kube-api-access-qs9c6\") pod \"a232efe8-1ff7-4370-8129-58adf4438cd8\" (UID: \"a232efe8-1ff7-4370-8129-58adf4438cd8\") "
Jul 31 02:24:17.191268 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.191142    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a232efe8-1ff7-4370-8129-58adf4438cd8-openshift-service-ca\") pod \"a232efe8-1ff7-4370-8129-58adf4438cd8\" (UID: \"a232efe8-1ff7-4370-8129-58adf4438cd8\") "
Jul 31 02:24:17.191330 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.191317    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lcng\" (UniqueName: \"kubernetes.io/projected/076c4581-b7a7-4b55-b2d7-e86be2214cba-kube-api-access-9lcng\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:17.191386 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.191333    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzpsp\" (UniqueName: \"kubernetes.io/projected/9f5884d2-6675-40bb-aab0-e9eb15604d96-kube-api-access-pzpsp\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:17.191386 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.191346    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076c4581-b7a7-4b55-b2d7-e86be2214cba-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:17.191386 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.191358    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5884d2-6675-40bb-aab0-e9eb15604d96-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:17.191582 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.191538    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a232efe8-1ff7-4370-8129-58adf4438cd8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a232efe8-1ff7-4370-8129-58adf4438cd8" (UID: "a232efe8-1ff7-4370-8129-58adf4438cd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:24:17.193061 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.193042    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a232efe8-1ff7-4370-8129-58adf4438cd8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a232efe8-1ff7-4370-8129-58adf4438cd8" (UID: "a232efe8-1ff7-4370-8129-58adf4438cd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:24:17.193116 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.193081    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a232efe8-1ff7-4370-8129-58adf4438cd8-kube-api-access-qs9c6" (OuterVolumeSpecName: "kube-api-access-qs9c6") pod "a232efe8-1ff7-4370-8129-58adf4438cd8" (UID: "a232efe8-1ff7-4370-8129-58adf4438cd8"). InnerVolumeSpecName "kube-api-access-qs9c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:24:17.257394 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.257374    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:17.292453 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.292430    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a232efe8-1ff7-4370-8129-58adf4438cd8-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:17.292453 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.292450    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs9c6\" (UniqueName: \"kubernetes.io/projected/a232efe8-1ff7-4370-8129-58adf4438cd8-kube-api-access-qs9c6\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:17.292597 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.292459    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a232efe8-1ff7-4370-8129-58adf4438cd8-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:17.552293 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.552209    2556 generic.go:356] "Generic (PLEG): container finished" podID="9f5884d2-6675-40bb-aab0-e9eb15604d96" containerID="73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c" exitCode=0
Jul 31 02:24:17.552293 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.552251    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh" event={"ID":"9f5884d2-6675-40bb-aab0-e9eb15604d96","Type":"ContainerDied","Data":"73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c"}
Jul 31 02:24:17.552293 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.552271    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"
Jul 31 02:24:17.552573 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.552300    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh" event={"ID":"9f5884d2-6675-40bb-aab0-e9eb15604d96","Type":"ContainerDied","Data":"ace1b03ffc42a8fdb15850d45e6cbcf7de82c6751b15a943a17750c617589536"}
Jul 31 02:24:17.552573 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.552322    2556 scope.go:117] "RemoveContainer" containerID="73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c"
Jul 31 02:24:17.553586 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.553543    2556 generic.go:356] "Generic (PLEG): container finished" podID="a232efe8-1ff7-4370-8129-58adf4438cd8" containerID="95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343" exitCode=0
Jul 31 02:24:17.553722 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.553596    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"
Jul 31 02:24:17.553722 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.553595    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s" event={"ID":"a232efe8-1ff7-4370-8129-58adf4438cd8","Type":"ContainerDied","Data":"95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343"}
Jul 31 02:24:17.553722 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.553642    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s" event={"ID":"a232efe8-1ff7-4370-8129-58adf4438cd8","Type":"ContainerDied","Data":"e4a48353e3043a3abf58be7d738471d378a42bde90d72bd0398f7811302f4fda"}
Jul 31 02:24:17.554976 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.554939    2556 generic.go:356] "Generic (PLEG): container finished" podID="076c4581-b7a7-4b55-b2d7-e86be2214cba" containerID="f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6" exitCode=0
Jul 31 02:24:17.555121 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.554998    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"
Jul 31 02:24:17.555121 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.555036    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58" event={"ID":"076c4581-b7a7-4b55-b2d7-e86be2214cba","Type":"ContainerDied","Data":"f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6"}
Jul 31 02:24:17.555121 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.555075    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58" event={"ID":"076c4581-b7a7-4b55-b2d7-e86be2214cba","Type":"ContainerDied","Data":"561c58ccd5f65790e011a79501bdcd4531dfdbd8e7fd195ab471a1dcc7c591f8"}
Jul 31 02:24:17.561313 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.561300    2556 scope.go:117] "RemoveContainer" containerID="73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c"
Jul 31 02:24:17.561607 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:24:17.561588    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c\": container with ID starting with 73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c not found: ID does not exist" containerID="73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c"
Jul 31 02:24:17.561672 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.561617    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c"} err="failed to get container status \"73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c\": rpc error: code = NotFound desc = could not find container \"73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c\": container with ID starting with 73b7f96b2daea62413ad7527cdd40f5db4427012472e2ebb49a184b2ec9c702c not found: ID does not exist"
Jul 31 02:24:17.561672 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.561633    2556 scope.go:117] "RemoveContainer" containerID="95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343"
Jul 31 02:24:17.569493 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.569478    2556 scope.go:117] "RemoveContainer" containerID="95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343"
Jul 31 02:24:17.569842 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:24:17.569824    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343\": container with ID starting with 95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343 not found: ID does not exist" containerID="95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343"
Jul 31 02:24:17.569914 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.569849    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343"} err="failed to get container status \"95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343\": rpc error: code = NotFound desc = could not find container \"95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343\": container with ID starting with 95268f2f2ec30fd6581c1314744f5b1182bd22fddca481fcd0b354d362764343 not found: ID does not exist"
Jul 31 02:24:17.569914 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.569863    2556 scope.go:117] "RemoveContainer" containerID="f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6"
Jul 31 02:24:17.576617 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.576597    2556 scope.go:117] "RemoveContainer" containerID="f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6"
Jul 31 02:24:17.576849 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:24:17.576834    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6\": container with ID starting with f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6 not found: ID does not exist" containerID="f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6"
Jul 31 02:24:17.576904 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.576855    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6"} err="failed to get container status \"f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6\": rpc error: code = NotFound desc = could not find container \"f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6\": container with ID starting with f94945f364bfead109177a159920bb83f3bc3d5835cead16f4ca6556f732cad6 not found: ID does not exist"
Jul 31 02:24:17.580115 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.580096    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"]
Jul 31 02:24:17.583402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.583381    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8af864-6bd8fdb5b4-99wbh"]
Jul 31 02:24:17.592877 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.592856    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"]
Jul 31 02:24:17.597016 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.597000    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8af864-5f47964dfb-4dc58"]
Jul 31 02:24:17.606396 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.606376    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"]
Jul 31 02:24:17.610076 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:17.610056    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8af864-54ff587b65-x5v5s"]
Jul 31 02:24:18.151947 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:18.151922    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="076c4581-b7a7-4b55-b2d7-e86be2214cba" path="/var/lib/kubelet/pods/076c4581-b7a7-4b55-b2d7-e86be2214cba/volumes"
Jul 31 02:24:18.152255 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:18.152199    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4330814c-58ee-44ff-befb-7ee65d481e1c" path="/var/lib/kubelet/pods/4330814c-58ee-44ff-befb-7ee65d481e1c/volumes"
Jul 31 02:24:18.152481 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:18.152469    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f5884d2-6675-40bb-aab0-e9eb15604d96" path="/var/lib/kubelet/pods/9f5884d2-6675-40bb-aab0-e9eb15604d96/volumes"
Jul 31 02:24:18.152805 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:18.152793    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a232efe8-1ff7-4370-8129-58adf4438cd8" path="/var/lib/kubelet/pods/a232efe8-1ff7-4370-8129-58adf4438cd8/volumes"
Jul 31 02:24:23.117629 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.117545    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:23.173658 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.173634    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:23.392334 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.392265    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:23.632220 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.632186    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:23.756753 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.756672    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp"]
Jul 31 02:24:23.757379 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.757355    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="076c4581-b7a7-4b55-b2d7-e86be2214cba" containerName="payload-processing"
Jul 31 02:24:23.757492 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.757381    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="076c4581-b7a7-4b55-b2d7-e86be2214cba" containerName="payload-processing"
Jul 31 02:24:23.757492 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.757407    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5884d2-6675-40bb-aab0-e9eb15604d96" containerName="payload-pre-processing"
Jul 31 02:24:23.757492 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.757415    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5884d2-6675-40bb-aab0-e9eb15604d96" containerName="payload-pre-processing"
Jul 31 02:24:23.757492 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.757437    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4330814c-58ee-44ff-befb-7ee65d481e1c" containerName="revoke-keys"
Jul 31 02:24:23.757492 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.757445    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4330814c-58ee-44ff-befb-7ee65d481e1c" containerName="revoke-keys"
Jul 31 02:24:23.757492 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.757467    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a232efe8-1ff7-4370-8129-58adf4438cd8" containerName="maas-api"
Jul 31 02:24:23.757492 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.757476    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a232efe8-1ff7-4370-8129-58adf4438cd8" containerName="maas-api"
Jul 31 02:24:23.757843 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.757628    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a232efe8-1ff7-4370-8129-58adf4438cd8" containerName="maas-api"
Jul 31 02:24:23.757843 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.757650    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="076c4581-b7a7-4b55-b2d7-e86be2214cba" containerName="payload-processing"
Jul 31 02:24:23.757843 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.757666    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4330814c-58ee-44ff-befb-7ee65d481e1c" containerName="revoke-keys"
Jul 31 02:24:23.757843 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.757676    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5884d2-6675-40bb-aab0-e9eb15604d96" containerName="payload-pre-processing"
Jul 31 02:24:23.763212 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.763187    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp"
Jul 31 02:24:23.763635 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.763611    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp"]
Jul 31 02:24:23.765353 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.765330    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:24:23.840399 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.840375    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8929efc4-5f95-4dab-8531-97f59c731a5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp\" (UID: \"8929efc4-5f95-4dab-8531-97f59c731a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp"
Jul 31 02:24:23.941719 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.941692    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8929efc4-5f95-4dab-8531-97f59c731a5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp\" (UID: \"8929efc4-5f95-4dab-8531-97f59c731a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp"
Jul 31 02:24:23.942236 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:23.942218    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8929efc4-5f95-4dab-8531-97f59c731a5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp\" (UID: \"8929efc4-5f95-4dab-8531-97f59c731a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp"
Jul 31 02:24:24.074192 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:24.074167    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp"
Jul 31 02:24:24.195805 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:24.195780    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp"]
Jul 31 02:24:24.199718 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:24:24.199693    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8929efc4_5f95_4dab_8531_97f59c731a5c.slice/crio-5eba7a450a2fa0f779a4c6372e8d9769e748ea73c4ba89f84652c9cc1fe1f766 WatchSource:0}: Error finding container 5eba7a450a2fa0f779a4c6372e8d9769e748ea73c4ba89f84652c9cc1fe1f766: Status 404 returned error can't find the container with id 5eba7a450a2fa0f779a4c6372e8d9769e748ea73c4ba89f84652c9cc1fe1f766
Jul 31 02:24:24.587084 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:24.587059    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp" event={"ID":"8929efc4-5f95-4dab-8531-97f59c731a5c","Type":"ContainerStarted","Data":"f93204a7f3a31ad4374f548a7f546b3b95bc0119b0090f34f93fa4604b92bdea"}
Jul 31 02:24:24.587207 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:24.587106    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp" event={"ID":"8929efc4-5f95-4dab-8531-97f59c731a5c","Type":"ContainerStarted","Data":"5eba7a450a2fa0f779a4c6372e8d9769e748ea73c4ba89f84652c9cc1fe1f766"}
Jul 31 02:24:24.601790 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:24.601743    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp" podStartSLOduration=1.601731015 podStartE2EDuration="1.601731015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:24:24.601173503 +0000 UTC m=+2822.966919128" watchObservedRunningTime="2026-07-31 02:24:24.601731015 +0000 UTC m=+2822.967476640"
Jul 31 02:24:25.592393 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:25.592357    2556 generic.go:356] "Generic (PLEG): container finished" podID="8929efc4-5f95-4dab-8531-97f59c731a5c" containerID="f93204a7f3a31ad4374f548a7f546b3b95bc0119b0090f34f93fa4604b92bdea" exitCode=0
Jul 31 02:24:25.592770 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:25.592441    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp" event={"ID":"8929efc4-5f95-4dab-8531-97f59c731a5c","Type":"ContainerDied","Data":"f93204a7f3a31ad4374f548a7f546b3b95bc0119b0090f34f93fa4604b92bdea"}
Jul 31 02:24:26.723184 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:26.723160    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp"
Jul 31 02:24:26.760534 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:26.760512    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8929efc4-5f95-4dab-8531-97f59c731a5c-maas-api-service-ca\") pod \"8929efc4-5f95-4dab-8531-97f59c731a5c\" (UID: \"8929efc4-5f95-4dab-8531-97f59c731a5c\") "
Jul 31 02:24:26.760804 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:26.760783    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8929efc4-5f95-4dab-8531-97f59c731a5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8929efc4-5f95-4dab-8531-97f59c731a5c" (UID: "8929efc4-5f95-4dab-8531-97f59c731a5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:24:26.861118 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:26.861069    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8929efc4-5f95-4dab-8531-97f59c731a5c-maas-api-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:27.602642 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:27.602619    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp"
Jul 31 02:24:27.602765 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:27.602665    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp" event={"ID":"8929efc4-5f95-4dab-8531-97f59c731a5c","Type":"ContainerDied","Data":"5eba7a450a2fa0f779a4c6372e8d9769e748ea73c4ba89f84652c9cc1fe1f766"}
Jul 31 02:24:27.602765 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:27.602707    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eba7a450a2fa0f779a4c6372e8d9769e748ea73c4ba89f84652c9cc1fe1f766"
Jul 31 02:24:32.805124 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:32.805097    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:34.114484 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.114452    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"]
Jul 31 02:24:34.114922 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.114677    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg" podUID="78130c40-1619-43ec-b442-745adc9266f1" containerName="maas-api" containerID="cri-o://c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6" gracePeriod=30
Jul 31 02:24:34.128287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.128264    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:34.153695 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.153668    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"]
Jul 31 02:24:34.154264 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.153932    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm" podUID="406f0c46-2c63-460a-9e65-923b450d2a57" containerName="payload-processing" containerID="cri-o://68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6" gracePeriod=30
Jul 31 02:24:34.157728 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.157595    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"]
Jul 31 02:24:34.157728 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.157789    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd" podUID="d55757cb-223a-44f5-bf06-bb61f945c426" containerName="payload-pre-processing" containerID="cri-o://93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752" gracePeriod=30
Jul 31 02:24:34.229573 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.229526    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:34.279678 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.279652    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp"]
Jul 31 02:24:34.282532 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.282508    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f88d9e-t4bdp"]
Jul 31 02:24:34.287403 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.287362    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:34.417580 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.416739    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:24:34.450372 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.450345    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"
Jul 31 02:24:34.475624 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.475603    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"
Jul 31 02:24:34.512823 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.512798    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z59vr\" (UniqueName: \"kubernetes.io/projected/406f0c46-2c63-460a-9e65-923b450d2a57-kube-api-access-z59vr\") pod \"406f0c46-2c63-460a-9e65-923b450d2a57\" (UID: \"406f0c46-2c63-460a-9e65-923b450d2a57\") "
Jul 31 02:24:34.512925 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.512832    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wdtw\" (UniqueName: \"kubernetes.io/projected/d55757cb-223a-44f5-bf06-bb61f945c426-kube-api-access-8wdtw\") pod \"d55757cb-223a-44f5-bf06-bb61f945c426\" (UID: \"d55757cb-223a-44f5-bf06-bb61f945c426\") "
Jul 31 02:24:34.512925 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.512859    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78130c40-1619-43ec-b442-745adc9266f1-maas-api-tls\") pod \"78130c40-1619-43ec-b442-745adc9266f1\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") "
Jul 31 02:24:34.512925 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.512882    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406f0c46-2c63-460a-9e65-923b450d2a57-plugins-config-volume\") pod \"406f0c46-2c63-460a-9e65-923b450d2a57\" (UID: \"406f0c46-2c63-460a-9e65-923b450d2a57\") "
Jul 31 02:24:34.513059 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.512924    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78130c40-1619-43ec-b442-745adc9266f1-openshift-service-ca\") pod \"78130c40-1619-43ec-b442-745adc9266f1\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") "
Jul 31 02:24:34.513059 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.512984    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cnhz\" (UniqueName: \"kubernetes.io/projected/78130c40-1619-43ec-b442-745adc9266f1-kube-api-access-7cnhz\") pod \"78130c40-1619-43ec-b442-745adc9266f1\" (UID: \"78130c40-1619-43ec-b442-745adc9266f1\") "
Jul 31 02:24:34.513059 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.513025    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55757cb-223a-44f5-bf06-bb61f945c426-plugins-config-volume\") pod \"d55757cb-223a-44f5-bf06-bb61f945c426\" (UID: \"d55757cb-223a-44f5-bf06-bb61f945c426\") "
Jul 31 02:24:34.513328 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.513303    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406f0c46-2c63-460a-9e65-923b450d2a57-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "406f0c46-2c63-460a-9e65-923b450d2a57" (UID: "406f0c46-2c63-460a-9e65-923b450d2a57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:24:34.513397 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.513310    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78130c40-1619-43ec-b442-745adc9266f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78130c40-1619-43ec-b442-745adc9266f1" (UID: "78130c40-1619-43ec-b442-745adc9266f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:24:34.513598 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.513579    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406f0c46-2c63-460a-9e65-923b450d2a57-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:34.513904 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.513888    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78130c40-1619-43ec-b442-745adc9266f1-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:34.513997 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.513948    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55757cb-223a-44f5-bf06-bb61f945c426-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d55757cb-223a-44f5-bf06-bb61f945c426" (UID: "d55757cb-223a-44f5-bf06-bb61f945c426"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:24:34.515116 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.515091    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78130c40-1619-43ec-b442-745adc9266f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78130c40-1619-43ec-b442-745adc9266f1" (UID: "78130c40-1619-43ec-b442-745adc9266f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:24:34.515226 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.515207    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78130c40-1619-43ec-b442-745adc9266f1-kube-api-access-7cnhz" (OuterVolumeSpecName: "kube-api-access-7cnhz") pod "78130c40-1619-43ec-b442-745adc9266f1" (UID: "78130c40-1619-43ec-b442-745adc9266f1"). InnerVolumeSpecName "kube-api-access-7cnhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:24:34.515270 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.515235    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406f0c46-2c63-460a-9e65-923b450d2a57-kube-api-access-z59vr" (OuterVolumeSpecName: "kube-api-access-z59vr") pod "406f0c46-2c63-460a-9e65-923b450d2a57" (UID: "406f0c46-2c63-460a-9e65-923b450d2a57"). InnerVolumeSpecName "kube-api-access-z59vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:24:34.515384 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.515370    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55757cb-223a-44f5-bf06-bb61f945c426-kube-api-access-8wdtw" (OuterVolumeSpecName: "kube-api-access-8wdtw") pod "d55757cb-223a-44f5-bf06-bb61f945c426" (UID: "d55757cb-223a-44f5-bf06-bb61f945c426"). InnerVolumeSpecName "kube-api-access-8wdtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:24:34.614528 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.614505    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wdtw\" (UniqueName: \"kubernetes.io/projected/d55757cb-223a-44f5-bf06-bb61f945c426-kube-api-access-8wdtw\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:34.614528 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.614526    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78130c40-1619-43ec-b442-745adc9266f1-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:34.614662 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.614535    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cnhz\" (UniqueName: \"kubernetes.io/projected/78130c40-1619-43ec-b442-745adc9266f1-kube-api-access-7cnhz\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:34.614662 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.614544    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55757cb-223a-44f5-bf06-bb61f945c426-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:34.614662 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.614551    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z59vr\" (UniqueName: \"kubernetes.io/projected/406f0c46-2c63-460a-9e65-923b450d2a57-kube-api-access-z59vr\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:24:34.638661 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.638614    2556 generic.go:356] "Generic (PLEG): container finished" podID="406f0c46-2c63-460a-9e65-923b450d2a57" containerID="68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6" exitCode=0
Jul 31 02:24:34.638661 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.638651    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"
Jul 31 02:24:34.638784 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.638706    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm" event={"ID":"406f0c46-2c63-460a-9e65-923b450d2a57","Type":"ContainerDied","Data":"68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6"}
Jul 31 02:24:34.638784 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.638776    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm" event={"ID":"406f0c46-2c63-460a-9e65-923b450d2a57","Type":"ContainerDied","Data":"fef7d9bd7f201a5959bc63e8eda7f83892cc457215a4096c5d9c7e7095ff4b07"}
Jul 31 02:24:34.638873 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.638800    2556 scope.go:117] "RemoveContainer" containerID="68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6"
Jul 31 02:24:34.640022 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.640007    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"
Jul 31 02:24:34.640108 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.639982    2556 generic.go:356] "Generic (PLEG): container finished" podID="d55757cb-223a-44f5-bf06-bb61f945c426" containerID="93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752" exitCode=0
Jul 31 02:24:34.640108 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.640053    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd" event={"ID":"d55757cb-223a-44f5-bf06-bb61f945c426","Type":"ContainerDied","Data":"93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752"}
Jul 31 02:24:34.640108 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.640091    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd" event={"ID":"d55757cb-223a-44f5-bf06-bb61f945c426","Type":"ContainerDied","Data":"bf74b9e91883a555da03680e008e493ad9c59aeb7c19d02dc314cdbd3351bfac"}
Jul 31 02:24:34.641691 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.641670    2556 generic.go:356] "Generic (PLEG): container finished" podID="78130c40-1619-43ec-b442-745adc9266f1" containerID="c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6" exitCode=0
Jul 31 02:24:34.641782 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.641696    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg" event={"ID":"78130c40-1619-43ec-b442-745adc9266f1","Type":"ContainerDied","Data":"c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6"}
Jul 31 02:24:34.641782 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.641736    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"
Jul 31 02:24:34.641862 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.641740    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg" event={"ID":"78130c40-1619-43ec-b442-745adc9266f1","Type":"ContainerDied","Data":"3e988b6d87aa47f4f808dbcd434f4560ba5955034b1312d58518585ade724975"}
Jul 31 02:24:34.647100 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.647079    2556 scope.go:117] "RemoveContainer" containerID="68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6"
Jul 31 02:24:34.647352 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:24:34.647327    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6\": container with ID starting with 68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6 not found: ID does not exist" containerID="68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6"
Jul 31 02:24:34.647502 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.647361    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6"} err="failed to get container status \"68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6\": rpc error: code = NotFound desc = could not find container \"68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6\": container with ID starting with 68018cf6f6fb1831c94f6c436413508e607eedded83bfc0dacddbee185e0e2d6 not found: ID does not exist"
Jul 31 02:24:34.647502 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.647381    2556 scope.go:117] "RemoveContainer" containerID="93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752"
Jul 31 02:24:34.655710 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.655696    2556 scope.go:117] "RemoveContainer" containerID="93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752"
Jul 31 02:24:34.655947 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:24:34.655928    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752\": container with ID starting with 93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752 not found: ID does not exist" containerID="93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752"
Jul 31 02:24:34.656009 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.655957    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752"} err="failed to get container status \"93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752\": rpc error: code = NotFound desc = could not find container \"93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752\": container with ID starting with 93777f3fbdfdfd42964aec74320cc10e6689bf06a3d86bad20fc395be1a28752 not found: ID does not exist"
Jul 31 02:24:34.656009 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.655979    2556 scope.go:117] "RemoveContainer" containerID="c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6"
Jul 31 02:24:34.663584 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.663542    2556 scope.go:117] "RemoveContainer" containerID="c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6"
Jul 31 02:24:34.663856 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:24:34.663837    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6\": container with ID starting with c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6 not found: ID does not exist" containerID="c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6"
Jul 31 02:24:34.663923 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.663869    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"]
Jul 31 02:24:34.663923 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.663865    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6"} err="failed to get container status \"c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6\": rpc error: code = NotFound desc = could not find container \"c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6\": container with ID starting with c1733c6cda36964351a56ff1854432cc2312f78c73d3b487fd9b0e145b1ed1e6 not found: ID does not exist"
Jul 31 02:24:34.666926 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.666907    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f88d9e-764f58dd76-b9xtd"]
Jul 31 02:24:34.676829 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.676808    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"]
Jul 31 02:24:34.680188 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.680171    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f88d9e-6cd68f9cb-kf5zm"]
Jul 31 02:24:34.689157 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.689138    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"]
Jul 31 02:24:34.689730 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.689713    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:34.695827 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:34.695806    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f88d9e-76dcc74bf-v77dg"]
Jul 31 02:24:36.150613 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:36.150589    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="406f0c46-2c63-460a-9e65-923b450d2a57" path="/var/lib/kubelet/pods/406f0c46-2c63-460a-9e65-923b450d2a57/volumes"
Jul 31 02:24:36.151045 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:36.151031    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78130c40-1619-43ec-b442-745adc9266f1" path="/var/lib/kubelet/pods/78130c40-1619-43ec-b442-745adc9266f1/volumes"
Jul 31 02:24:36.151297 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:36.151286    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8929efc4-5f95-4dab-8531-97f59c731a5c" path="/var/lib/kubelet/pods/8929efc4-5f95-4dab-8531-97f59c731a5c/volumes"
Jul 31 02:24:36.151581 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:36.151570    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d55757cb-223a-44f5-bf06-bb61f945c426" path="/var/lib/kubelet/pods/d55757cb-223a-44f5-bf06-bb61f945c426/volumes"
Jul 31 02:24:36.408052 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:36.407998    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:40.680635 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:40.680603    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:40.741096 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:40.741070    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:40.943431 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:40.943361    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:41.168518 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:41.168492    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:41.522673 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:41.522651    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:41.836654 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:41.836624    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:41.992515 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:41.992490    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:44.442005 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:44.441971    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:44.541233 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:44.541201    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:45.520722 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.520695    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"]
Jul 31 02:24:45.521177 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.520987    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78130c40-1619-43ec-b442-745adc9266f1" containerName="maas-api"
Jul 31 02:24:45.521177 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.520997    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="78130c40-1619-43ec-b442-745adc9266f1" containerName="maas-api"
Jul 31 02:24:45.521177 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.521007    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8929efc4-5f95-4dab-8531-97f59c731a5c" containerName="revoke-keys"
Jul 31 02:24:45.521177 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.521012    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8929efc4-5f95-4dab-8531-97f59c731a5c" containerName="revoke-keys"
Jul 31 02:24:45.521177 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.521017    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d55757cb-223a-44f5-bf06-bb61f945c426" containerName="payload-pre-processing"
Jul 31 02:24:45.521177 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.521024    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d55757cb-223a-44f5-bf06-bb61f945c426" containerName="payload-pre-processing"
Jul 31 02:24:45.521177 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.521032    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="406f0c46-2c63-460a-9e65-923b450d2a57" containerName="payload-processing"
Jul 31 02:24:45.521177 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.521036    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="406f0c46-2c63-460a-9e65-923b450d2a57" containerName="payload-processing"
Jul 31 02:24:45.521177 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.521086    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="406f0c46-2c63-460a-9e65-923b450d2a57" containerName="payload-processing"
Jul 31 02:24:45.521177 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.521095    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d55757cb-223a-44f5-bf06-bb61f945c426" containerName="payload-pre-processing"
Jul 31 02:24:45.521177 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.521102    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8929efc4-5f95-4dab-8531-97f59c731a5c" containerName="revoke-keys"
Jul 31 02:24:45.521177 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.521107    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="78130c40-1619-43ec-b442-745adc9266f1" containerName="maas-api"
Jul 31 02:24:45.525610 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.525591    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:45.527618 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.527600    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bf4f62-serving-cert\""
Jul 31 02:24:45.538539 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.538519    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"]
Jul 31 02:24:45.561187 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.561160    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:45.589787 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.589763    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6e705a-c18e-43ed-a496-88304ba7f714-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:45.589890 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.589800    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6e705a-c18e-43ed-a496-88304ba7f714-maas-api-tls\") pod \"maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:45.589890 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.589841    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkd7w\" (UniqueName: \"kubernetes.io/projected/4a6e705a-c18e-43ed-a496-88304ba7f714-kube-api-access-xkd7w\") pod \"maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:45.690810 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.690787    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6e705a-c18e-43ed-a496-88304ba7f714-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:45.690918 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.690814    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6e705a-c18e-43ed-a496-88304ba7f714-maas-api-tls\") pod \"maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:45.690918 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.690840    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkd7w\" (UniqueName: \"kubernetes.io/projected/4a6e705a-c18e-43ed-a496-88304ba7f714-kube-api-access-xkd7w\") pod \"maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:45.691040 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:24:45.690936    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-serving-cert: secret "maas-api-e2e-api-a-bf4f62-serving-cert" not found
Jul 31 02:24:45.691040 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:24:45.691009    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6e705a-c18e-43ed-a496-88304ba7f714-maas-api-tls podName:4a6e705a-c18e-43ed-a496-88304ba7f714 nodeName:}" failed. No retries permitted until 2026-07-31 02:24:46.190992392 +0000 UTC m=+2844.556737997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a6e705a-c18e-43ed-a496-88304ba7f714-maas-api-tls") pod "maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v" (UID: "4a6e705a-c18e-43ed-a496-88304ba7f714") : secret "maas-api-e2e-api-a-bf4f62-serving-cert" not found
Jul 31 02:24:45.691441 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.691422    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6e705a-c18e-43ed-a496-88304ba7f714-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:45.698496 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.698479    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkd7w\" (UniqueName: \"kubernetes.io/projected/4a6e705a-c18e-43ed-a496-88304ba7f714-kube-api-access-xkd7w\") pod \"maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:45.791998 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:45.791943    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:46.166253 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.166220    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:46.193968 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.193938    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6e705a-c18e-43ed-a496-88304ba7f714-maas-api-tls\") pod \"maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:46.196418 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.196391    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6e705a-c18e-43ed-a496-88304ba7f714-maas-api-tls\") pod \"maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:46.260674 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.260644    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:24:46.282685 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.282660    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"]
Jul 31 02:24:46.286042 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.286024    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"
Jul 31 02:24:46.288370 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.288349    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bf4f62-dockercfg-mfhj6\""
Jul 31 02:24:46.288472 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.288413    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bf4f62\""
Jul 31 02:24:46.296707 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.296685    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"]
Jul 31 02:24:46.338295 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.338270    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"]
Jul 31 02:24:46.341407 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.341393    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"
Jul 31 02:24:46.349367 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.349346    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"]
Jul 31 02:24:46.395300 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.395273    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47fvn\" (UniqueName: \"kubernetes.io/projected/ac95bf32-97cb-4f21-8a27-3b80523b3c98-kube-api-access-47fvn\") pod \"payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8\" (UID: \"ac95bf32-97cb-4f21-8a27-3b80523b3c98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"
Jul 31 02:24:46.395476 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.395313    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950b51f7-de26-4bae-83ba-b17e3f7e2d4a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7\" (UID: \"950b51f7-de26-4bae-83ba-b17e3f7e2d4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"
Jul 31 02:24:46.395476 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.395442    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2np7\" (UniqueName: \"kubernetes.io/projected/950b51f7-de26-4bae-83ba-b17e3f7e2d4a-kube-api-access-w2np7\") pod \"payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7\" (UID: \"950b51f7-de26-4bae-83ba-b17e3f7e2d4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"
Jul 31 02:24:46.395685 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.395661    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac95bf32-97cb-4f21-8a27-3b80523b3c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8\" (UID: \"ac95bf32-97cb-4f21-8a27-3b80523b3c98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"
Jul 31 02:24:46.435459 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.435396    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:46.496973 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.496947    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47fvn\" (UniqueName: \"kubernetes.io/projected/ac95bf32-97cb-4f21-8a27-3b80523b3c98-kube-api-access-47fvn\") pod \"payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8\" (UID: \"ac95bf32-97cb-4f21-8a27-3b80523b3c98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"
Jul 31 02:24:46.497112 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.496984    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950b51f7-de26-4bae-83ba-b17e3f7e2d4a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7\" (UID: \"950b51f7-de26-4bae-83ba-b17e3f7e2d4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"
Jul 31 02:24:46.497112 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.497029    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2np7\" (UniqueName: \"kubernetes.io/projected/950b51f7-de26-4bae-83ba-b17e3f7e2d4a-kube-api-access-w2np7\") pod \"payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7\" (UID: \"950b51f7-de26-4bae-83ba-b17e3f7e2d4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"
Jul 31 02:24:46.497112 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.497057    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac95bf32-97cb-4f21-8a27-3b80523b3c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8\" (UID: \"ac95bf32-97cb-4f21-8a27-3b80523b3c98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"
Jul 31 02:24:46.497779 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.497753    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950b51f7-de26-4bae-83ba-b17e3f7e2d4a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7\" (UID: \"950b51f7-de26-4bae-83ba-b17e3f7e2d4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"
Jul 31 02:24:46.497892 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.497760    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac95bf32-97cb-4f21-8a27-3b80523b3c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8\" (UID: \"ac95bf32-97cb-4f21-8a27-3b80523b3c98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"
Jul 31 02:24:46.505133 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.505104    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2np7\" (UniqueName: \"kubernetes.io/projected/950b51f7-de26-4bae-83ba-b17e3f7e2d4a-kube-api-access-w2np7\") pod \"payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7\" (UID: \"950b51f7-de26-4bae-83ba-b17e3f7e2d4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"
Jul 31 02:24:46.505693 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.505666    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47fvn\" (UniqueName: \"kubernetes.io/projected/ac95bf32-97cb-4f21-8a27-3b80523b3c98-kube-api-access-47fvn\") pod \"payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8\" (UID: \"ac95bf32-97cb-4f21-8a27-3b80523b3c98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"
Jul 31 02:24:46.568409 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.568381    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"]
Jul 31 02:24:46.571289 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:24:46.571267    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6e705a_c18e_43ed_a496_88304ba7f714.slice/crio-0c67e5a8143127dd35b41b443196ec8447d0dad2c0fe769aaae08dc454ffce66 WatchSource:0}: Error finding container 0c67e5a8143127dd35b41b443196ec8447d0dad2c0fe769aaae08dc454ffce66: Status 404 returned error can't find the container with id 0c67e5a8143127dd35b41b443196ec8447d0dad2c0fe769aaae08dc454ffce66
Jul 31 02:24:46.572452 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.572435    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 02:24:46.596036 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.596015    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"
Jul 31 02:24:46.651537 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.651509    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"
Jul 31 02:24:46.696453 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.696328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v" event={"ID":"4a6e705a-c18e-43ed-a496-88304ba7f714","Type":"ContainerStarted","Data":"0c67e5a8143127dd35b41b443196ec8447d0dad2c0fe769aaae08dc454ffce66"}
Jul 31 02:24:46.723750 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.723725    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"]
Jul 31 02:24:46.727684 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:24:46.727659    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950b51f7_de26_4bae_83ba_b17e3f7e2d4a.slice/crio-906d3086ce2d92907a5a68bb4e04dda404b72fe72b8cf5ec541fdc5a13eb5fa3 WatchSource:0}: Error finding container 906d3086ce2d92907a5a68bb4e04dda404b72fe72b8cf5ec541fdc5a13eb5fa3: Status 404 returned error can't find the container with id 906d3086ce2d92907a5a68bb4e04dda404b72fe72b8cf5ec541fdc5a13eb5fa3
Jul 31 02:24:46.788660 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:46.788634    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"]
Jul 31 02:24:46.792007 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:24:46.791978    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac95bf32_97cb_4f21_8a27_3b80523b3c98.slice/crio-2029d76c4feb7e811b377b008265dd6151640d44ccfb580f569045d5aef48826 WatchSource:0}: Error finding container 2029d76c4feb7e811b377b008265dd6151640d44ccfb580f569045d5aef48826: Status 404 returned error can't find the container with id 2029d76c4feb7e811b377b008265dd6151640d44ccfb580f569045d5aef48826
Jul 31 02:24:47.703579 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:47.703480    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v" event={"ID":"4a6e705a-c18e-43ed-a496-88304ba7f714","Type":"ContainerStarted","Data":"50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513"}
Jul 31 02:24:47.703993 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:47.703614    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:47.705547 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:47.705512    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8" event={"ID":"ac95bf32-97cb-4f21-8a27-3b80523b3c98","Type":"ContainerStarted","Data":"7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf"}
Jul 31 02:24:47.705677 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:47.705594    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8" event={"ID":"ac95bf32-97cb-4f21-8a27-3b80523b3c98","Type":"ContainerStarted","Data":"2029d76c4feb7e811b377b008265dd6151640d44ccfb580f569045d5aef48826"}
Jul 31 02:24:47.705950 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:47.705928    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"
Jul 31 02:24:47.707858 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:47.707839    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7" event={"ID":"950b51f7-de26-4bae-83ba-b17e3f7e2d4a","Type":"ContainerStarted","Data":"e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a"}
Jul 31 02:24:47.707970 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:47.707872    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7" event={"ID":"950b51f7-de26-4bae-83ba-b17e3f7e2d4a","Type":"ContainerStarted","Data":"906d3086ce2d92907a5a68bb4e04dda404b72fe72b8cf5ec541fdc5a13eb5fa3"}
Jul 31 02:24:47.708058 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:47.707970    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"
Jul 31 02:24:47.720238 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:47.720200    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v" podStartSLOduration=2.188064139 podStartE2EDuration="2.720186975s" totalImagesPullingTime="532.122836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:24:47.718728699 +0000 UTC m=+2846.084474327" watchObservedRunningTime="2026-07-31 02:24:47.720186975 +0000 UTC m=+2846.085932600"
Jul 31 02:24:47.732673 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:47.732631    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7" podStartSLOduration=1.732616173 podStartE2EDuration="1.732616173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:24:47.731581819 +0000 UTC m=+2846.097327445" watchObservedRunningTime="2026-07-31 02:24:47.732616173 +0000 UTC m=+2846.098361799"
Jul 31 02:24:47.745926 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:47.745886    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8" podStartSLOduration=1.745872183 podStartE2EDuration="1.745872183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:24:47.745830807 +0000 UTC m=+2846.111576437" watchObservedRunningTime="2026-07-31 02:24:47.745872183 +0000 UTC m=+2846.111617808"
Jul 31 02:24:53.717743 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:53.717714    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:24:58.714386 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:58.714358    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"
Jul 31 02:24:58.714791 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:24:58.714451    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"
Jul 31 02:25:14.578600 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:14.578506    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:25:15.072715 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:15.072685    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:25:15.384840 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:15.384774    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:25:15.840917 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:15.840890    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:25:18.615720 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:18.615688    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:25:18.715944 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:18.715916    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:25:19.718262 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.718234    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"]
Jul 31 02:25:19.722140 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.722116    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:19.725082 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.725064    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ba3761-serving-cert\""
Jul 31 02:25:19.735242 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.735219    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"]
Jul 31 02:25:19.775849 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.775829    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:25:19.843003 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.842980    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-openshift-service-ca\") pod \"maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:19.843111 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.843028    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brnm9\" (UniqueName: \"kubernetes.io/projected/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-kube-api-access-brnm9\") pod \"maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:19.843111 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.843071    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-maas-api-tls\") pod \"maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:19.943797 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.943768    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-maas-api-tls\") pod \"maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:19.943918 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.943842    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-openshift-service-ca\") pod \"maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:19.943918 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.943899    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brnm9\" (UniqueName: \"kubernetes.io/projected/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-kube-api-access-brnm9\") pod \"maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:19.943918 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:25:19.943904    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-serving-cert: secret "maas-api-e2e-api-b-ba3761-serving-cert" not found
Jul 31 02:25:19.944075 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:25:19.943969    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-maas-api-tls podName:93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f nodeName:}" failed. No retries permitted until 2026-07-31 02:25:20.443952199 +0000 UTC m=+2878.809697804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-maas-api-tls") pod "maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57" (UID: "93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f") : secret "maas-api-e2e-api-b-ba3761-serving-cert" not found
Jul 31 02:25:19.944420 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.944402    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-openshift-service-ca\") pod \"maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:19.955861 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:19.955843    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brnm9\" (UniqueName: \"kubernetes.io/projected/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-kube-api-access-brnm9\") pod \"maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:20.215655 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.215629    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:25:20.318376 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.318348    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:25:20.401803 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.401774    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:25:20.402478 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.402459    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"]
Jul 31 02:25:20.406012 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.405993    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"
Jul 31 02:25:20.408287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.408269    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ba3761\""
Jul 31 02:25:20.408287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.408282    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ba3761-dockercfg-7njh2\""
Jul 31 02:25:20.419899 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.419879    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"]
Jul 31 02:25:20.441393 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.441368    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"]
Jul 31 02:25:20.444658 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.444645    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"
Jul 31 02:25:20.448674 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.448655    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-maas-api-tls\") pod \"maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:20.451580 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.451496    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-maas-api-tls\") pod \"maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:20.452219 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.451996    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"]
Jul 31 02:25:20.549299 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.549229    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3957c2c8-078f-4779-a4db-4b8069268e33-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv\" (UID: \"3957c2c8-078f-4779-a4db-4b8069268e33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"
Jul 31 02:25:20.549299 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.549264    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfssg\" (UniqueName: \"kubernetes.io/projected/3957c2c8-078f-4779-a4db-4b8069268e33-kube-api-access-kfssg\") pod \"payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv\" (UID: \"3957c2c8-078f-4779-a4db-4b8069268e33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"
Jul 31 02:25:20.549299 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.549289    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f180c425-803f-4f60-a959-5776b4fb3fae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ba3761-89d6b594-5grkj\" (UID: \"f180c425-803f-4f60-a959-5776b4fb3fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"
Jul 31 02:25:20.549496 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.549311    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdqfm\" (UniqueName: \"kubernetes.io/projected/f180c425-803f-4f60-a959-5776b4fb3fae-kube-api-access-xdqfm\") pod \"payload-processing-e2e-api-b-ba3761-89d6b594-5grkj\" (UID: \"f180c425-803f-4f60-a959-5776b4fb3fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"
Jul 31 02:25:20.631852 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.631829    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:20.649909 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.649884    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfssg\" (UniqueName: \"kubernetes.io/projected/3957c2c8-078f-4779-a4db-4b8069268e33-kube-api-access-kfssg\") pod \"payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv\" (UID: \"3957c2c8-078f-4779-a4db-4b8069268e33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"
Jul 31 02:25:20.650027 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.649928    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f180c425-803f-4f60-a959-5776b4fb3fae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ba3761-89d6b594-5grkj\" (UID: \"f180c425-803f-4f60-a959-5776b4fb3fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"
Jul 31 02:25:20.650027 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.649954    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdqfm\" (UniqueName: \"kubernetes.io/projected/f180c425-803f-4f60-a959-5776b4fb3fae-kube-api-access-xdqfm\") pod \"payload-processing-e2e-api-b-ba3761-89d6b594-5grkj\" (UID: \"f180c425-803f-4f60-a959-5776b4fb3fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"
Jul 31 02:25:20.650139 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.650054    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3957c2c8-078f-4779-a4db-4b8069268e33-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv\" (UID: \"3957c2c8-078f-4779-a4db-4b8069268e33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"
Jul 31 02:25:20.650743 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.650719    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f180c425-803f-4f60-a959-5776b4fb3fae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ba3761-89d6b594-5grkj\" (UID: \"f180c425-803f-4f60-a959-5776b4fb3fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"
Jul 31 02:25:20.650743 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.650740    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3957c2c8-078f-4779-a4db-4b8069268e33-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv\" (UID: \"3957c2c8-078f-4779-a4db-4b8069268e33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"
Jul 31 02:25:20.656951 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.656932    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdqfm\" (UniqueName: \"kubernetes.io/projected/f180c425-803f-4f60-a959-5776b4fb3fae-kube-api-access-xdqfm\") pod \"payload-processing-e2e-api-b-ba3761-89d6b594-5grkj\" (UID: \"f180c425-803f-4f60-a959-5776b4fb3fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"
Jul 31 02:25:20.657211 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.657182    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfssg\" (UniqueName: \"kubernetes.io/projected/3957c2c8-078f-4779-a4db-4b8069268e33-kube-api-access-kfssg\") pod \"payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv\" (UID: \"3957c2c8-078f-4779-a4db-4b8069268e33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"
Jul 31 02:25:20.715958 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.715929    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"
Jul 31 02:25:20.759075 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.759041    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"]
Jul 31 02:25:20.762760 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:25:20.762727    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ecd9ae_af57_47a3_a4f7_ebbd49dd3f5f.slice/crio-649b6159dd8882828816375449b8c71c35da824943e5925e20274fea189be4ae WatchSource:0}: Error finding container 649b6159dd8882828816375449b8c71c35da824943e5925e20274fea189be4ae: Status 404 returned error can't find the container with id 649b6159dd8882828816375449b8c71c35da824943e5925e20274fea189be4ae
Jul 31 02:25:20.763726 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.763707    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"
Jul 31 02:25:20.858138 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:20.858108    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57" event={"ID":"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f","Type":"ContainerStarted","Data":"649b6159dd8882828816375449b8c71c35da824943e5925e20274fea189be4ae"}
Jul 31 02:25:21.061315 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.061287    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"]
Jul 31 02:25:21.063628 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:25:21.063600    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3957c2c8_078f_4779_a4db_4b8069268e33.slice/crio-1ffde85b0c5b52e2db092155f9b4108310667f13e004b0fb5eb5345326b888b2 WatchSource:0}: Error finding container 1ffde85b0c5b52e2db092155f9b4108310667f13e004b0fb5eb5345326b888b2: Status 404 returned error can't find the container with id 1ffde85b0c5b52e2db092155f9b4108310667f13e004b0fb5eb5345326b888b2
Jul 31 02:25:21.114140 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.114116    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"]
Jul 31 02:25:21.120651 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:25:21.120623    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf180c425_803f_4f60_a959_5776b4fb3fae.slice/crio-e874956bb8874e8ea4daafebfbfeb725f79873d445c194cd8daed3f6221da661 WatchSource:0}: Error finding container e874956bb8874e8ea4daafebfbfeb725f79873d445c194cd8daed3f6221da661: Status 404 returned error can't find the container with id e874956bb8874e8ea4daafebfbfeb725f79873d445c194cd8daed3f6221da661
Jul 31 02:25:21.863666 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.863636    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv" event={"ID":"3957c2c8-078f-4779-a4db-4b8069268e33","Type":"ContainerStarted","Data":"535df49024ca53f61fdd613e84251953b92f5783dd4647efcadbe9637118b90a"}
Jul 31 02:25:21.864068 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.863682    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv" event={"ID":"3957c2c8-078f-4779-a4db-4b8069268e33","Type":"ContainerStarted","Data":"1ffde85b0c5b52e2db092155f9b4108310667f13e004b0fb5eb5345326b888b2"}
Jul 31 02:25:21.864068 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.863727    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"
Jul 31 02:25:21.865281 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.865258    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57" event={"ID":"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f","Type":"ContainerStarted","Data":"c528ad608d9cc3d8e417ba28393200f310bcf53a1509f098b4a1350b073c9d0c"}
Jul 31 02:25:21.865412 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.865358    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:21.866846 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.866818    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj" event={"ID":"f180c425-803f-4f60-a959-5776b4fb3fae","Type":"ContainerStarted","Data":"ba2906470bbae2d024021a312b077993338ba82f2ba980537cbd23c31b96f6f6"}
Jul 31 02:25:21.866961 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.866857    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj" event={"ID":"f180c425-803f-4f60-a959-5776b4fb3fae","Type":"ContainerStarted","Data":"e874956bb8874e8ea4daafebfbfeb725f79873d445c194cd8daed3f6221da661"}
Jul 31 02:25:21.866961 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.866941    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"
Jul 31 02:25:21.879731 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.879695    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv" podStartSLOduration=1.8796835870000002 podStartE2EDuration="1.879683587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:25:21.877971351 +0000 UTC m=+2880.243716977" watchObservedRunningTime="2026-07-31 02:25:21.879683587 +0000 UTC m=+2880.245429232"
Jul 31 02:25:21.894582 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.894527    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57" podStartSLOduration=2.450297585 podStartE2EDuration="2.894511241s" totalImagesPullingTime="444.213656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:25:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:25:21.892317897 +0000 UTC m=+2880.258063519" watchObservedRunningTime="2026-07-31 02:25:21.894511241 +0000 UTC m=+2880.260256873"
Jul 31 02:25:21.906895 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:21.906860    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj" podStartSLOduration=1.906850012 podStartE2EDuration="1.906850012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:25:21.905821012 +0000 UTC m=+2880.271566634" watchObservedRunningTime="2026-07-31 02:25:21.906850012 +0000 UTC m=+2880.272595637"
Jul 31 02:25:22.879447 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:22.879341    2556 scope.go:117] "RemoveContainer" containerID="32ffb7bca5b4b8ec83f12c248c9d01aa1514ffcafc2336957e1aa741e163208e"
Jul 31 02:25:27.876458 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:27.876427    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:25:32.873473 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:32.873442    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"
Jul 31 02:25:32.873940 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:32.873612    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"
Jul 31 02:25:43.714247 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:25:43.714210    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:12.714163 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:12.714133    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"]
Jul 31 02:26:12.714743 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:12.714369    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v" podUID="4a6e705a-c18e-43ed-a496-88304ba7f714" containerName="maas-api" containerID="cri-o://50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513" gracePeriod=30
Jul 31 02:26:12.749608 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:12.749553    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:12.765467 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:12.765441    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"]
Jul 31 02:26:12.765678 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:12.765655    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7" podUID="950b51f7-de26-4bae-83ba-b17e3f7e2d4a" containerName="payload-processing" containerID="cri-o://e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a" gracePeriod=30
Jul 31 02:26:12.773357 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:12.773336    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"]
Jul 31 02:26:12.773651 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:12.773530    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8" podUID="ac95bf32-97cb-4f21-8a27-3b80523b3c98" containerName="payload-pre-processing" containerID="cri-o://7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf" gracePeriod=30
Jul 31 02:26:12.849535 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:12.849513    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:12.905723 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:12.905650    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:13.019624 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.019600    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"
Jul 31 02:26:13.052402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.052289    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:26:13.059061 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.059041    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"
Jul 31 02:26:13.104332 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.104307    2556 generic.go:356] "Generic (PLEG): container finished" podID="4a6e705a-c18e-43ed-a496-88304ba7f714" containerID="50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513" exitCode=0
Jul 31 02:26:13.104476 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.104348    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"
Jul 31 02:26:13.104476 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.104385    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v" event={"ID":"4a6e705a-c18e-43ed-a496-88304ba7f714","Type":"ContainerDied","Data":"50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513"}
Jul 31 02:26:13.104476 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.104434    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v" event={"ID":"4a6e705a-c18e-43ed-a496-88304ba7f714","Type":"ContainerDied","Data":"0c67e5a8143127dd35b41b443196ec8447d0dad2c0fe769aaae08dc454ffce66"}
Jul 31 02:26:13.104476 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.104462    2556 scope.go:117] "RemoveContainer" containerID="50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513"
Jul 31 02:26:13.105662 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.105639    2556 generic.go:356] "Generic (PLEG): container finished" podID="ac95bf32-97cb-4f21-8a27-3b80523b3c98" containerID="7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf" exitCode=0
Jul 31 02:26:13.105750 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.105670    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8" event={"ID":"ac95bf32-97cb-4f21-8a27-3b80523b3c98","Type":"ContainerDied","Data":"7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf"}
Jul 31 02:26:13.105750 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.105688    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"
Jul 31 02:26:13.105750 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.105726    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8" event={"ID":"ac95bf32-97cb-4f21-8a27-3b80523b3c98","Type":"ContainerDied","Data":"2029d76c4feb7e811b377b008265dd6151640d44ccfb580f569045d5aef48826"}
Jul 31 02:26:13.107157 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.107136    2556 generic.go:356] "Generic (PLEG): container finished" podID="950b51f7-de26-4bae-83ba-b17e3f7e2d4a" containerID="e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a" exitCode=0
Jul 31 02:26:13.107266 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.107196    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"
Jul 31 02:26:13.107266 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.107216    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7" event={"ID":"950b51f7-de26-4bae-83ba-b17e3f7e2d4a","Type":"ContainerDied","Data":"e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a"}
Jul 31 02:26:13.107266 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.107255    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7" event={"ID":"950b51f7-de26-4bae-83ba-b17e3f7e2d4a","Type":"ContainerDied","Data":"906d3086ce2d92907a5a68bb4e04dda404b72fe72b8cf5ec541fdc5a13eb5fa3"}
Jul 31 02:26:13.113096 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.113081    2556 scope.go:117] "RemoveContainer" containerID="50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513"
Jul 31 02:26:13.113348 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:26:13.113332    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513\": container with ID starting with 50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513 not found: ID does not exist" containerID="50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513"
Jul 31 02:26:13.113395 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.113357    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513"} err="failed to get container status \"50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513\": rpc error: code = NotFound desc = could not find container \"50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513\": container with ID starting with 50288af1dd185d20b219dffdcc93250d29969aaeaa86aa8b5b8eb41161251513 not found: ID does not exist"
Jul 31 02:26:13.113395 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.113373    2556 scope.go:117] "RemoveContainer" containerID="7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf"
Jul 31 02:26:13.120484 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.120469    2556 scope.go:117] "RemoveContainer" containerID="7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf"
Jul 31 02:26:13.120736 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:26:13.120720    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf\": container with ID starting with 7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf not found: ID does not exist" containerID="7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf"
Jul 31 02:26:13.120799 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.120740    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf"} err="failed to get container status \"7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf\": rpc error: code = NotFound desc = could not find container \"7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf\": container with ID starting with 7f9b0448d819d4e41a49c6a3f511dc5a894f5644de2b08c62dbeb7999a7f4adf not found: ID does not exist"
Jul 31 02:26:13.120799 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.120753    2556 scope.go:117] "RemoveContainer" containerID="e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a"
Jul 31 02:26:13.125865 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.125845    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47fvn\" (UniqueName: \"kubernetes.io/projected/ac95bf32-97cb-4f21-8a27-3b80523b3c98-kube-api-access-47fvn\") pod \"ac95bf32-97cb-4f21-8a27-3b80523b3c98\" (UID: \"ac95bf32-97cb-4f21-8a27-3b80523b3c98\") "
Jul 31 02:26:13.125939 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.125887    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6e705a-c18e-43ed-a496-88304ba7f714-maas-api-tls\") pod \"4a6e705a-c18e-43ed-a496-88304ba7f714\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") "
Jul 31 02:26:13.126083 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.126067    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2np7\" (UniqueName: \"kubernetes.io/projected/950b51f7-de26-4bae-83ba-b17e3f7e2d4a-kube-api-access-w2np7\") pod \"950b51f7-de26-4bae-83ba-b17e3f7e2d4a\" (UID: \"950b51f7-de26-4bae-83ba-b17e3f7e2d4a\") "
Jul 31 02:26:13.126123 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.126101    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950b51f7-de26-4bae-83ba-b17e3f7e2d4a-plugins-config-volume\") pod \"950b51f7-de26-4bae-83ba-b17e3f7e2d4a\" (UID: \"950b51f7-de26-4bae-83ba-b17e3f7e2d4a\") "
Jul 31 02:26:13.126168 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.126130    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6e705a-c18e-43ed-a496-88304ba7f714-openshift-service-ca\") pod \"4a6e705a-c18e-43ed-a496-88304ba7f714\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") "
Jul 31 02:26:13.126168 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.126155    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac95bf32-97cb-4f21-8a27-3b80523b3c98-plugins-config-volume\") pod \"ac95bf32-97cb-4f21-8a27-3b80523b3c98\" (UID: \"ac95bf32-97cb-4f21-8a27-3b80523b3c98\") "
Jul 31 02:26:13.126830 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.126319    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkd7w\" (UniqueName: \"kubernetes.io/projected/4a6e705a-c18e-43ed-a496-88304ba7f714-kube-api-access-xkd7w\") pod \"4a6e705a-c18e-43ed-a496-88304ba7f714\" (UID: \"4a6e705a-c18e-43ed-a496-88304ba7f714\") "
Jul 31 02:26:13.126830 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.126476    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6e705a-c18e-43ed-a496-88304ba7f714-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a6e705a-c18e-43ed-a496-88304ba7f714" (UID: "4a6e705a-c18e-43ed-a496-88304ba7f714"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:26:13.126830 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.126482    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac95bf32-97cb-4f21-8a27-3b80523b3c98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac95bf32-97cb-4f21-8a27-3b80523b3c98" (UID: "ac95bf32-97cb-4f21-8a27-3b80523b3c98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:26:13.126830 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.126656    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6e705a-c18e-43ed-a496-88304ba7f714-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:13.126830 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.126674    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac95bf32-97cb-4f21-8a27-3b80523b3c98-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:13.126830 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.126681    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950b51f7-de26-4bae-83ba-b17e3f7e2d4a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "950b51f7-de26-4bae-83ba-b17e3f7e2d4a" (UID: "950b51f7-de26-4bae-83ba-b17e3f7e2d4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:26:13.128021 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.128000    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6e705a-c18e-43ed-a496-88304ba7f714-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a6e705a-c18e-43ed-a496-88304ba7f714" (UID: "4a6e705a-c18e-43ed-a496-88304ba7f714"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:26:13.128097 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.128014    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac95bf32-97cb-4f21-8a27-3b80523b3c98-kube-api-access-47fvn" (OuterVolumeSpecName: "kube-api-access-47fvn") pod "ac95bf32-97cb-4f21-8a27-3b80523b3c98" (UID: "ac95bf32-97cb-4f21-8a27-3b80523b3c98"). InnerVolumeSpecName "kube-api-access-47fvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:26:13.128151 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.128135    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950b51f7-de26-4bae-83ba-b17e3f7e2d4a-kube-api-access-w2np7" (OuterVolumeSpecName: "kube-api-access-w2np7") pod "950b51f7-de26-4bae-83ba-b17e3f7e2d4a" (UID: "950b51f7-de26-4bae-83ba-b17e3f7e2d4a"). InnerVolumeSpecName "kube-api-access-w2np7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:26:13.128874 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.128856    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6e705a-c18e-43ed-a496-88304ba7f714-kube-api-access-xkd7w" (OuterVolumeSpecName: "kube-api-access-xkd7w") pod "4a6e705a-c18e-43ed-a496-88304ba7f714" (UID: "4a6e705a-c18e-43ed-a496-88304ba7f714"). InnerVolumeSpecName "kube-api-access-xkd7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:26:13.129071 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.129050    2556 scope.go:117] "RemoveContainer" containerID="e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a"
Jul 31 02:26:13.129293 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:26:13.129273    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a\": container with ID starting with e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a not found: ID does not exist" containerID="e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a"
Jul 31 02:26:13.129332 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.129299    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a"} err="failed to get container status \"e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a\": rpc error: code = NotFound desc = could not find container \"e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a\": container with ID starting with e34b1865e0ec1efe843a31ca2a414a039a3b7acdbf442571fb7b45a2e2b23f8a not found: ID does not exist"
Jul 31 02:26:13.227978 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.227926    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47fvn\" (UniqueName: \"kubernetes.io/projected/ac95bf32-97cb-4f21-8a27-3b80523b3c98-kube-api-access-47fvn\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:13.227978 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.227949    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6e705a-c18e-43ed-a496-88304ba7f714-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:13.227978 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.227958    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2np7\" (UniqueName: \"kubernetes.io/projected/950b51f7-de26-4bae-83ba-b17e3f7e2d4a-kube-api-access-w2np7\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:13.227978 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.227967    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950b51f7-de26-4bae-83ba-b17e3f7e2d4a-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:13.227978 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.227975    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkd7w\" (UniqueName: \"kubernetes.io/projected/4a6e705a-c18e-43ed-a496-88304ba7f714-kube-api-access-xkd7w\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:13.317873 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.317855    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:13.432237 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.432216    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"]
Jul 31 02:26:13.435517 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.435498    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf4f62-6d975b5b8-8mgx7"]
Jul 31 02:26:13.445315 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.445298    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"]
Jul 31 02:26:13.449295 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.449278    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf4f62-7df7846757-q8b6v"]
Jul 31 02:26:13.457934 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.457916    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"]
Jul 31 02:26:13.465587 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:13.462226    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf4f62-7885f8c6b8-prjs8"]
Jul 31 02:26:14.151049 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:14.151014    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a6e705a-c18e-43ed-a496-88304ba7f714" path="/var/lib/kubelet/pods/4a6e705a-c18e-43ed-a496-88304ba7f714/volumes"
Jul 31 02:26:14.151381 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:14.151366    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="950b51f7-de26-4bae-83ba-b17e3f7e2d4a" path="/var/lib/kubelet/pods/950b51f7-de26-4bae-83ba-b17e3f7e2d4a/volumes"
Jul 31 02:26:14.151716 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:14.151703    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac95bf32-97cb-4f21-8a27-3b80523b3c98" path="/var/lib/kubelet/pods/ac95bf32-97cb-4f21-8a27-3b80523b3c98/volumes"
Jul 31 02:26:18.946779 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:18.946749    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:19.049082 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.049057    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:19.238686 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.238601    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:19.500776 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.500716    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:19.631042 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.631018    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f"]
Jul 31 02:26:19.631347 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.631335    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a6e705a-c18e-43ed-a496-88304ba7f714" containerName="maas-api"
Jul 31 02:26:19.631388 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.631348    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a6e705a-c18e-43ed-a496-88304ba7f714" containerName="maas-api"
Jul 31 02:26:19.631388 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.631359    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="950b51f7-de26-4bae-83ba-b17e3f7e2d4a" containerName="payload-processing"
Jul 31 02:26:19.631388 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.631365    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="950b51f7-de26-4bae-83ba-b17e3f7e2d4a" containerName="payload-processing"
Jul 31 02:26:19.631388 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.631374    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac95bf32-97cb-4f21-8a27-3b80523b3c98" containerName="payload-pre-processing"
Jul 31 02:26:19.631388 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.631379    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac95bf32-97cb-4f21-8a27-3b80523b3c98" containerName="payload-pre-processing"
Jul 31 02:26:19.631537 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.631434    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="950b51f7-de26-4bae-83ba-b17e3f7e2d4a" containerName="payload-processing"
Jul 31 02:26:19.631537 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.631443    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac95bf32-97cb-4f21-8a27-3b80523b3c98" containerName="payload-pre-processing"
Jul 31 02:26:19.631537 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.631449    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a6e705a-c18e-43ed-a496-88304ba7f714" containerName="maas-api"
Jul 31 02:26:19.635613 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.635598    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f"
Jul 31 02:26:19.637798 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.637778    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:26:19.641006 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.640927    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f"]
Jul 31 02:26:19.674713 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.674686    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85056ed4-bcfc-4a7a-8346-745b9a339c25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f\" (UID: \"85056ed4-bcfc-4a7a-8346-745b9a339c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f"
Jul 31 02:26:19.775166 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.775111    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85056ed4-bcfc-4a7a-8346-745b9a339c25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f\" (UID: \"85056ed4-bcfc-4a7a-8346-745b9a339c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f"
Jul 31 02:26:19.775671 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.775655    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85056ed4-bcfc-4a7a-8346-745b9a339c25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f\" (UID: \"85056ed4-bcfc-4a7a-8346-745b9a339c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f"
Jul 31 02:26:19.946790 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:19.946768    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f"
Jul 31 02:26:20.070066 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:20.070036    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f"]
Jul 31 02:26:20.072841 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:26:20.072818    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85056ed4_bcfc_4a7a_8346_745b9a339c25.slice/crio-fdfa1450fe0c13a242e7f91296ae923a41d53fb7edf8f85133d0148cb947c19e WatchSource:0}: Error finding container fdfa1450fe0c13a242e7f91296ae923a41d53fb7edf8f85133d0148cb947c19e: Status 404 returned error can't find the container with id fdfa1450fe0c13a242e7f91296ae923a41d53fb7edf8f85133d0148cb947c19e
Jul 31 02:26:20.144772 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:20.144738    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f" event={"ID":"85056ed4-bcfc-4a7a-8346-745b9a339c25","Type":"ContainerStarted","Data":"fdfa1450fe0c13a242e7f91296ae923a41d53fb7edf8f85133d0148cb947c19e"}
Jul 31 02:26:21.149943 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:21.149915    2556 generic.go:356] "Generic (PLEG): container finished" podID="85056ed4-bcfc-4a7a-8346-745b9a339c25" containerID="681a04c5af02da7c7f03ab52b652c485b8c96cbf52c53505e515c6a8da442a32" exitCode=0
Jul 31 02:26:21.150293 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:21.150005    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f" event={"ID":"85056ed4-bcfc-4a7a-8346-745b9a339c25","Type":"ContainerDied","Data":"681a04c5af02da7c7f03ab52b652c485b8c96cbf52c53505e515c6a8da442a32"}
Jul 31 02:26:22.279673 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:22.279647    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f"
Jul 31 02:26:22.396365 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:22.396342    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85056ed4-bcfc-4a7a-8346-745b9a339c25-maas-api-service-ca\") pod \"85056ed4-bcfc-4a7a-8346-745b9a339c25\" (UID: \"85056ed4-bcfc-4a7a-8346-745b9a339c25\") "
Jul 31 02:26:22.396659 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:22.396640    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85056ed4-bcfc-4a7a-8346-745b9a339c25-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85056ed4-bcfc-4a7a-8346-745b9a339c25" (UID: "85056ed4-bcfc-4a7a-8346-745b9a339c25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:26:22.496992 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:22.496942    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85056ed4-bcfc-4a7a-8346-745b9a339c25-maas-api-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:22.910519 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:22.910497    2556 scope.go:117] "RemoveContainer" containerID="4ea6f70ecf7f34804c5f0b061b448f400f3d4a3a9e31a296a9a943a3d198af92"
Jul 31 02:26:23.158614 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:23.158583    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f" event={"ID":"85056ed4-bcfc-4a7a-8346-745b9a339c25","Type":"ContainerDied","Data":"fdfa1450fe0c13a242e7f91296ae923a41d53fb7edf8f85133d0148cb947c19e"}
Jul 31 02:26:23.158699 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:23.158616    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f"
Jul 31 02:26:23.158699 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:23.158625    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdfa1450fe0c13a242e7f91296ae923a41d53fb7edf8f85133d0148cb947c19e"
Jul 31 02:26:34.996412 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:34.996384    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"]
Jul 31 02:26:34.996829 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:34.996577    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57" podUID="93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f" containerName="maas-api" containerID="cri-o://c528ad608d9cc3d8e417ba28393200f310bcf53a1509f098b4a1350b073c9d0c" gracePeriod=30
Jul 31 02:26:35.023322 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.023301    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:35.048124 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.048098    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"]
Jul 31 02:26:35.048339 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.048315    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj" podUID="f180c425-803f-4f60-a959-5776b4fb3fae" containerName="payload-processing" containerID="cri-o://ba2906470bbae2d024021a312b077993338ba82f2ba980537cbd23c31b96f6f6" gracePeriod=30
Jul 31 02:26:35.056142 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.056119    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"]
Jul 31 02:26:35.056418 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.056394    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv" podUID="3957c2c8-078f-4779-a4db-4b8069268e33" containerName="payload-pre-processing" containerID="cri-o://535df49024ca53f61fdd613e84251953b92f5783dd4647efcadbe9637118b90a" gracePeriod=30
Jul 31 02:26:35.174824 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.174776    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f"]
Jul 31 02:26:35.176405 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.176383    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba3761-9lz4f"]
Jul 31 02:26:35.196128 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.196095    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:35.214454 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.214419    2556 generic.go:356] "Generic (PLEG): container finished" podID="3957c2c8-078f-4779-a4db-4b8069268e33" containerID="535df49024ca53f61fdd613e84251953b92f5783dd4647efcadbe9637118b90a" exitCode=0
Jul 31 02:26:35.214454 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.214448    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv" event={"ID":"3957c2c8-078f-4779-a4db-4b8069268e33","Type":"ContainerDied","Data":"535df49024ca53f61fdd613e84251953b92f5783dd4647efcadbe9637118b90a"}
Jul 31 02:26:35.216401 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.216377    2556 generic.go:356] "Generic (PLEG): container finished" podID="93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f" containerID="c528ad608d9cc3d8e417ba28393200f310bcf53a1509f098b4a1350b073c9d0c" exitCode=0
Jul 31 02:26:35.216584 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.216488    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57" event={"ID":"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f","Type":"ContainerDied","Data":"c528ad608d9cc3d8e417ba28393200f310bcf53a1509f098b4a1350b073c9d0c"}
Jul 31 02:26:35.218088 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.218067    2556 generic.go:356] "Generic (PLEG): container finished" podID="f180c425-803f-4f60-a959-5776b4fb3fae" containerID="ba2906470bbae2d024021a312b077993338ba82f2ba980537cbd23c31b96f6f6" exitCode=0
Jul 31 02:26:35.218194 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.218146    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj" event={"ID":"f180c425-803f-4f60-a959-5776b4fb3fae","Type":"ContainerDied","Data":"ba2906470bbae2d024021a312b077993338ba82f2ba980537cbd23c31b96f6f6"}
Jul 31 02:26:35.252303 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.252234    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:35.311307 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.311286    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"
Jul 31 02:26:35.346191 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.346172    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"
Jul 31 02:26:35.350275 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.350256    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:26:35.387332 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.387311    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brnm9\" (UniqueName: \"kubernetes.io/projected/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-kube-api-access-brnm9\") pod \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") "
Jul 31 02:26:35.387451 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.387339    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdqfm\" (UniqueName: \"kubernetes.io/projected/f180c425-803f-4f60-a959-5776b4fb3fae-kube-api-access-xdqfm\") pod \"f180c425-803f-4f60-a959-5776b4fb3fae\" (UID: \"f180c425-803f-4f60-a959-5776b4fb3fae\") "
Jul 31 02:26:35.387451 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.387386    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-maas-api-tls\") pod \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") "
Jul 31 02:26:35.387451 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.387417    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3957c2c8-078f-4779-a4db-4b8069268e33-plugins-config-volume\") pod \"3957c2c8-078f-4779-a4db-4b8069268e33\" (UID: \"3957c2c8-078f-4779-a4db-4b8069268e33\") "
Jul 31 02:26:35.387451 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.387444    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f180c425-803f-4f60-a959-5776b4fb3fae-plugins-config-volume\") pod \"f180c425-803f-4f60-a959-5776b4fb3fae\" (UID: \"f180c425-803f-4f60-a959-5776b4fb3fae\") "
Jul 31 02:26:35.387706 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.387497    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfssg\" (UniqueName: \"kubernetes.io/projected/3957c2c8-078f-4779-a4db-4b8069268e33-kube-api-access-kfssg\") pod \"3957c2c8-078f-4779-a4db-4b8069268e33\" (UID: \"3957c2c8-078f-4779-a4db-4b8069268e33\") "
Jul 31 02:26:35.387706 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.387526    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-openshift-service-ca\") pod \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\" (UID: \"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f\") "
Jul 31 02:26:35.388054 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.388019    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f180c425-803f-4f60-a959-5776b4fb3fae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f180c425-803f-4f60-a959-5776b4fb3fae" (UID: "f180c425-803f-4f60-a959-5776b4fb3fae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:26:35.388169 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.388065    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f" (UID: "93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:26:35.388169 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.388078    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3957c2c8-078f-4779-a4db-4b8069268e33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3957c2c8-078f-4779-a4db-4b8069268e33" (UID: "3957c2c8-078f-4779-a4db-4b8069268e33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:26:35.389715 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.389689    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f180c425-803f-4f60-a959-5776b4fb3fae-kube-api-access-xdqfm" (OuterVolumeSpecName: "kube-api-access-xdqfm") pod "f180c425-803f-4f60-a959-5776b4fb3fae" (UID: "f180c425-803f-4f60-a959-5776b4fb3fae"). InnerVolumeSpecName "kube-api-access-xdqfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:26:35.390065 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.390044    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f" (UID: "93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:26:35.390065 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.390052    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3957c2c8-078f-4779-a4db-4b8069268e33-kube-api-access-kfssg" (OuterVolumeSpecName: "kube-api-access-kfssg") pod "3957c2c8-078f-4779-a4db-4b8069268e33" (UID: "3957c2c8-078f-4779-a4db-4b8069268e33"). InnerVolumeSpecName "kube-api-access-kfssg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:26:35.390207 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.390189    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-kube-api-access-brnm9" (OuterVolumeSpecName: "kube-api-access-brnm9") pod "93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f" (UID: "93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f"). InnerVolumeSpecName "kube-api-access-brnm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:26:35.489103 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.489084    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f180c425-803f-4f60-a959-5776b4fb3fae-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:35.489103 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.489104    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfssg\" (UniqueName: \"kubernetes.io/projected/3957c2c8-078f-4779-a4db-4b8069268e33-kube-api-access-kfssg\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:35.489237 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.489112    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:35.489237 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.489121    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brnm9\" (UniqueName: \"kubernetes.io/projected/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-kube-api-access-brnm9\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:35.489237 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.489129    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdqfm\" (UniqueName: \"kubernetes.io/projected/f180c425-803f-4f60-a959-5776b4fb3fae-kube-api-access-xdqfm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:35.489237 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.489137    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:35.489237 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.489146    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3957c2c8-078f-4779-a4db-4b8069268e33-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:26:35.590731 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:35.590708    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:36.150828 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.150804    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85056ed4-bcfc-4a7a-8346-745b9a339c25" path="/var/lib/kubelet/pods/85056ed4-bcfc-4a7a-8346-745b9a339c25/volumes"
Jul 31 02:26:36.222680 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.222665    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"
Jul 31 02:26:36.222776 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.222660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv" event={"ID":"3957c2c8-078f-4779-a4db-4b8069268e33","Type":"ContainerDied","Data":"1ffde85b0c5b52e2db092155f9b4108310667f13e004b0fb5eb5345326b888b2"}
Jul 31 02:26:36.222817 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.222786    2556 scope.go:117] "RemoveContainer" containerID="535df49024ca53f61fdd613e84251953b92f5783dd4647efcadbe9637118b90a"
Jul 31 02:26:36.224049 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.223984    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"
Jul 31 02:26:36.224049 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.224010    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57" event={"ID":"93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f","Type":"ContainerDied","Data":"649b6159dd8882828816375449b8c71c35da824943e5925e20274fea189be4ae"}
Jul 31 02:26:36.231467 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.231447    2556 scope.go:117] "RemoveContainer" containerID="c528ad608d9cc3d8e417ba28393200f310bcf53a1509f098b4a1350b073c9d0c"
Jul 31 02:26:36.231640 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.231618    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj" event={"ID":"f180c425-803f-4f60-a959-5776b4fb3fae","Type":"ContainerDied","Data":"e874956bb8874e8ea4daafebfbfeb725f79873d445c194cd8daed3f6221da661"}
Jul 31 02:26:36.231718 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.231645    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"
Jul 31 02:26:36.240363 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.240277    2556 scope.go:117] "RemoveContainer" containerID="ba2906470bbae2d024021a312b077993338ba82f2ba980537cbd23c31b96f6f6"
Jul 31 02:26:36.240842 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.240787    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"]
Jul 31 02:26:36.242502 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.242480    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba3761-68bdb666fd-j6c57"]
Jul 31 02:26:36.251169 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.251148    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"]
Jul 31 02:26:36.252785 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.252769    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ba3761-667c56f84-8l5nv"]
Jul 31 02:26:36.262519 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.262478    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"]
Jul 31 02:26:36.265961 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:36.265945    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ba3761-89d6b594-5grkj"]
Jul 31 02:26:38.150839 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:38.150809    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3957c2c8-078f-4779-a4db-4b8069268e33" path="/var/lib/kubelet/pods/3957c2c8-078f-4779-a4db-4b8069268e33/volumes"
Jul 31 02:26:38.151192 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:38.151147    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f" path="/var/lib/kubelet/pods/93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f/volumes"
Jul 31 02:26:38.151426 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:38.151413    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f180c425-803f-4f60-a959-5776b4fb3fae" path="/var/lib/kubelet/pods/f180c425-803f-4f60-a959-5776b4fb3fae/volumes"
Jul 31 02:26:41.387851 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:41.387824    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:41.488190 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:41.488164    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:41.793252 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:41.793182    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:42.000964 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:42.000935    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:42.356620 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:42.356593    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:42.684093 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:42.684004    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:42.765848 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:42.765779    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:46.178098 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:46.178064    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:46.273386 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:46.273356    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:47.413426 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413391    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"]
Jul 31 02:26:47.413828 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413709    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3957c2c8-078f-4779-a4db-4b8069268e33" containerName="payload-pre-processing"
Jul 31 02:26:47.413828 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413726    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3957c2c8-078f-4779-a4db-4b8069268e33" containerName="payload-pre-processing"
Jul 31 02:26:47.413828 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413741    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85056ed4-bcfc-4a7a-8346-745b9a339c25" containerName="revoke-keys"
Jul 31 02:26:47.413828 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413749    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="85056ed4-bcfc-4a7a-8346-745b9a339c25" containerName="revoke-keys"
Jul 31 02:26:47.413828 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413758    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f180c425-803f-4f60-a959-5776b4fb3fae" containerName="payload-processing"
Jul 31 02:26:47.413828 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413767    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f180c425-803f-4f60-a959-5776b4fb3fae" containerName="payload-processing"
Jul 31 02:26:47.413828 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413780    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f" containerName="maas-api"
Jul 31 02:26:47.413828 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413785    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f" containerName="maas-api"
Jul 31 02:26:47.414074 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413842    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="85056ed4-bcfc-4a7a-8346-745b9a339c25" containerName="revoke-keys"
Jul 31 02:26:47.414074 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413849    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ecd9ae-af57-47a3-a4f7-ebbd49dd3f5f" containerName="maas-api"
Jul 31 02:26:47.414074 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413854    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f180c425-803f-4f60-a959-5776b4fb3fae" containerName="payload-processing"
Jul 31 02:26:47.414074 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.413861    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3957c2c8-078f-4779-a4db-4b8069268e33" containerName="payload-pre-processing"
Jul 31 02:26:47.418598 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.418553    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:26:47.420649 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.420626    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-2aed0b-serving-cert\""
Jul 31 02:26:47.428502 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.428470    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"]
Jul 31 02:26:47.445836 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.445817    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:47.475900 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.475877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ce3593-7284-47f1-8546-f7f401a7a9f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs\" (UID: \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:26:47.475987 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.475906    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ce3593-7284-47f1-8546-f7f401a7a9f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs\" (UID: \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:26:47.475987 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.475922    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csgr5\" (UniqueName: \"kubernetes.io/projected/e4ce3593-7284-47f1-8546-f7f401a7a9f0-kube-api-access-csgr5\") pod \"maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs\" (UID: \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:26:47.576513 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.576481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ce3593-7284-47f1-8546-f7f401a7a9f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs\" (UID: \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:26:47.576687 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.576518    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ce3593-7284-47f1-8546-f7f401a7a9f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs\" (UID: \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:26:47.576687 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.576533    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgr5\" (UniqueName: \"kubernetes.io/projected/e4ce3593-7284-47f1-8546-f7f401a7a9f0-kube-api-access-csgr5\") pod \"maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs\" (UID: \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:26:47.577279 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.577246    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ce3593-7284-47f1-8546-f7f401a7a9f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs\" (UID: \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:26:47.577402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.577382    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:47.579025 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.579010    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ce3593-7284-47f1-8546-f7f401a7a9f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs\" (UID: \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:26:47.585336 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.585308    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgr5\" (UniqueName: \"kubernetes.io/projected/e4ce3593-7284-47f1-8546-f7f401a7a9f0-kube-api-access-csgr5\") pod \"maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs\" (UID: \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:26:47.730787 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.730738    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:26:47.857200 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.857170    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"]
Jul 31 02:26:47.860228 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:26:47.860171    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ce3593_7284_47f1_8546_f7f401a7a9f0.slice/crio-a5e8166c11ff6ac84a11fe90c3260f6d1cb44375864a786793c2f1d36eac3cb2 WatchSource:0}: Error finding container a5e8166c11ff6ac84a11fe90c3260f6d1cb44375864a786793c2f1d36eac3cb2: Status 404 returned error can't find the container with id a5e8166c11ff6ac84a11fe90c3260f6d1cb44375864a786793c2f1d36eac3cb2
Jul 31 02:26:47.900600 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.900577    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:47.990073 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.990015    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"]
Jul 31 02:26:47.994445 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.994429    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"
Jul 31 02:26:47.997224 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.997202    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-2aed0b\""
Jul 31 02:26:47.997302 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.997211    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-2aed0b-dockercfg-m9sk9\""
Jul 31 02:26:47.999510 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:47.999489    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:26:48.005964 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.005941    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"]
Jul 31 02:26:48.026967 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.026943    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"]
Jul 31 02:26:48.030069 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.030056    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"
Jul 31 02:26:48.038797 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.038775    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"]
Jul 31 02:26:48.081012 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.080986    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnnpp\" (UniqueName: \"kubernetes.io/projected/8e2b2759-8e22-4886-ab27-429992953694-kube-api-access-jnnpp\") pod \"payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m\" (UID: \"8e2b2759-8e22-4886-ab27-429992953694\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"
Jul 31 02:26:48.081120 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.081021    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps\" (UID: \"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"
Jul 31 02:26:48.081120 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.081045    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knff9\" (UniqueName: \"kubernetes.io/projected/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff-kube-api-access-knff9\") pod \"payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps\" (UID: \"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"
Jul 31 02:26:48.081200 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.081120    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b2759-8e22-4886-ab27-429992953694-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m\" (UID: \"8e2b2759-8e22-4886-ab27-429992953694\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"
Jul 31 02:26:48.182410 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.182377    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnnpp\" (UniqueName: \"kubernetes.io/projected/8e2b2759-8e22-4886-ab27-429992953694-kube-api-access-jnnpp\") pod \"payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m\" (UID: \"8e2b2759-8e22-4886-ab27-429992953694\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"
Jul 31 02:26:48.182410 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.182413    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps\" (UID: \"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"
Jul 31 02:26:48.182661 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.182433    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knff9\" (UniqueName: \"kubernetes.io/projected/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff-kube-api-access-knff9\") pod \"payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps\" (UID: \"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"
Jul 31 02:26:48.182661 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.182549    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b2759-8e22-4886-ab27-429992953694-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m\" (UID: \"8e2b2759-8e22-4886-ab27-429992953694\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"
Jul 31 02:26:48.183096 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.183076    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps\" (UID: \"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"
Jul 31 02:26:48.183193 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.183169    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b2759-8e22-4886-ab27-429992953694-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m\" (UID: \"8e2b2759-8e22-4886-ab27-429992953694\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"
Jul 31 02:26:48.195259 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.195238    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnnpp\" (UniqueName: \"kubernetes.io/projected/8e2b2759-8e22-4886-ab27-429992953694-kube-api-access-jnnpp\") pod \"payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m\" (UID: \"8e2b2759-8e22-4886-ab27-429992953694\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"
Jul 31 02:26:48.195889 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.195871    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knff9\" (UniqueName: \"kubernetes.io/projected/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff-kube-api-access-knff9\") pod \"payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps\" (UID: \"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"
Jul 31 02:26:48.284799 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.284744    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs" event={"ID":"e4ce3593-7284-47f1-8546-f7f401a7a9f0","Type":"ContainerStarted","Data":"a5e8166c11ff6ac84a11fe90c3260f6d1cb44375864a786793c2f1d36eac3cb2"}
Jul 31 02:26:48.304637 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.304618    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"
Jul 31 02:26:48.340884 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.340853    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"
Jul 31 02:26:48.443755 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.443723    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"]
Jul 31 02:26:48.445551 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:26:48.445527    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2b2759_8e22_4886_ab27_429992953694.slice/crio-ea2df13c27294c8c82fb06ef9101a7d78035704c688caa158d4b79471e2cd46d WatchSource:0}: Error finding container ea2df13c27294c8c82fb06ef9101a7d78035704c688caa158d4b79471e2cd46d: Status 404 returned error can't find the container with id ea2df13c27294c8c82fb06ef9101a7d78035704c688caa158d4b79471e2cd46d
Jul 31 02:26:48.485350 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:48.485255    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"]
Jul 31 02:26:48.487604 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:26:48.487575    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5d2a85_4ea5_423d_9a84_ec340fbe73ff.slice/crio-0e27be60ee47830e972a47418f82b976e68ebdc996edaa75c7a229ba6ef9a4f7 WatchSource:0}: Error finding container 0e27be60ee47830e972a47418f82b976e68ebdc996edaa75c7a229ba6ef9a4f7: Status 404 returned error can't find the container with id 0e27be60ee47830e972a47418f82b976e68ebdc996edaa75c7a229ba6ef9a4f7
Jul 31 02:26:49.290487 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:49.290450    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m" event={"ID":"8e2b2759-8e22-4886-ab27-429992953694","Type":"ContainerStarted","Data":"8934b3091233e8a7035d165cb742d990b32ca9d761b20df6878298d7edee5503"}
Jul 31 02:26:49.290729 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:49.290518    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m" event={"ID":"8e2b2759-8e22-4886-ab27-429992953694","Type":"ContainerStarted","Data":"ea2df13c27294c8c82fb06ef9101a7d78035704c688caa158d4b79471e2cd46d"}
Jul 31 02:26:49.290729 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:49.290680    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"
Jul 31 02:26:49.292067 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:49.292043    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs" event={"ID":"e4ce3593-7284-47f1-8546-f7f401a7a9f0","Type":"ContainerStarted","Data":"96b8a25d44a09bcf751024c58d1440c4f7aed7f41bf452831887c8e8acbeee5e"}
Jul 31 02:26:49.292184 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:49.292108    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:26:49.293628 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:49.293610    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps" event={"ID":"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff","Type":"ContainerStarted","Data":"e9e29ec3a2edddc86df14bb63b262d22273a9eaadaac79fdaeda5b42c6a8ba5a"}
Jul 31 02:26:49.293705 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:49.293641    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps" event={"ID":"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff","Type":"ContainerStarted","Data":"0e27be60ee47830e972a47418f82b976e68ebdc996edaa75c7a229ba6ef9a4f7"}
Jul 31 02:26:49.293760 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:49.293732    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"
Jul 31 02:26:49.304947 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:49.304910    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m" podStartSLOduration=2.304898771 podStartE2EDuration="2.304898771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:26:49.30419669 +0000 UTC m=+2967.669942311" watchObservedRunningTime="2026-07-31 02:26:49.304898771 +0000 UTC m=+2967.670644396"
Jul 31 02:26:49.319295 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:49.319256    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs" podStartSLOduration=1.8949391260000001 podStartE2EDuration="2.319245787s" totalImagesPullingTime="424.306661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:26:49.318692182 +0000 UTC m=+2967.684437802" watchObservedRunningTime="2026-07-31 02:26:49.319245787 +0000 UTC m=+2967.684991404"
Jul 31 02:26:49.338014 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:49.337862    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps" podStartSLOduration=1.337848214 podStartE2EDuration="1.337848214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:26:49.337505877 +0000 UTC m=+2967.703251503" watchObservedRunningTime="2026-07-31 02:26:49.337848214 +0000 UTC m=+2967.703593841"
Jul 31 02:26:55.303975 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:26:55.303949    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:27:00.300951 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:00.300917    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"
Jul 31 02:27:00.301798 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:00.301771    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"
Jul 31 02:27:11.278508 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:11.278473    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:11.646117 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:11.646085    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:11.978016 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:11.977945    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:12.583917 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:12.583888    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:15.343282 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:15.343255    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:15.444867 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:15.444820    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:16.420378 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.420341    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"]
Jul 31 02:27:16.424358 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.424340    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:16.426455 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.426432    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-01a44c-serving-cert\""
Jul 31 02:27:16.431728 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.431706    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"]
Jul 31 02:27:16.450755 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.450734    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:16.589615 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.589592    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wzxf\" (UniqueName: \"kubernetes.io/projected/054b0c8e-37b7-4837-8b3d-5a6a422ed171-kube-api-access-9wzxf\") pod \"maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc\" (UID: \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:16.589739 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.589630    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054b0c8e-37b7-4837-8b3d-5a6a422ed171-maas-api-tls\") pod \"maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc\" (UID: \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:16.589739 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.589661    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054b0c8e-37b7-4837-8b3d-5a6a422ed171-openshift-service-ca\") pod \"maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc\" (UID: \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:16.622815 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.622791    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:16.690146 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.690083    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wzxf\" (UniqueName: \"kubernetes.io/projected/054b0c8e-37b7-4837-8b3d-5a6a422ed171-kube-api-access-9wzxf\") pod \"maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc\" (UID: \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:16.690146 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.690125    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054b0c8e-37b7-4837-8b3d-5a6a422ed171-maas-api-tls\") pod \"maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc\" (UID: \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:16.690294 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.690167    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054b0c8e-37b7-4837-8b3d-5a6a422ed171-openshift-service-ca\") pod \"maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc\" (UID: \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:16.690804 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.690784    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054b0c8e-37b7-4837-8b3d-5a6a422ed171-openshift-service-ca\") pod \"maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc\" (UID: \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:16.692499 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.692472    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054b0c8e-37b7-4837-8b3d-5a6a422ed171-maas-api-tls\") pod \"maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc\" (UID: \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:16.698014 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.697985    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wzxf\" (UniqueName: \"kubernetes.io/projected/054b0c8e-37b7-4837-8b3d-5a6a422ed171-kube-api-access-9wzxf\") pod \"maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc\" (UID: \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:16.735315 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.735290    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:16.859331 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.859299    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"]
Jul 31 02:27:16.861332 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:27:16.861305    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054b0c8e_37b7_4837_8b3d_5a6a422ed171.slice/crio-b4a5a7a7fa1a7ed792267ce1cc42879bdabb855a691d18e980356d891e078c16 WatchSource:0}: Error finding container b4a5a7a7fa1a7ed792267ce1cc42879bdabb855a691d18e980356d891e078c16: Status 404 returned error can't find the container with id b4a5a7a7fa1a7ed792267ce1cc42879bdabb855a691d18e980356d891e078c16
Jul 31 02:27:16.954856 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:16.954797    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:17.050112 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.050090    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:17.079709 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.079687    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"]
Jul 31 02:27:17.084138 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.084123    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"
Jul 31 02:27:17.086397 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.086378    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-01a44c-dockercfg-xjhks\""
Jul 31 02:27:17.086483 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.086378    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-01a44c\""
Jul 31 02:27:17.093671 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.093650    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"]
Jul 31 02:27:17.122782 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.122759    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"]
Jul 31 02:27:17.125886 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.125869    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"
Jul 31 02:27:17.133456 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.133437    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"]
Jul 31 02:27:17.194652 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.194624    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2drzd\" (UniqueName: \"kubernetes.io/projected/92de2058-3d2c-4b20-b78f-cff131643350-kube-api-access-2drzd\") pod \"payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9\" (UID: \"92de2058-3d2c-4b20-b78f-cff131643350\") " pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"
Jul 31 02:27:17.194796 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.194655    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92de2058-3d2c-4b20-b78f-cff131643350-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9\" (UID: \"92de2058-3d2c-4b20-b78f-cff131643350\") " pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"
Jul 31 02:27:17.295494 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.295473    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4511df-478e-4cda-9b83-e6fac0490d3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n\" (UID: \"cf4511df-478e-4cda-9b83-e6fac0490d3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"
Jul 31 02:27:17.295633 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.295528    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2drzd\" (UniqueName: \"kubernetes.io/projected/92de2058-3d2c-4b20-b78f-cff131643350-kube-api-access-2drzd\") pod \"payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9\" (UID: \"92de2058-3d2c-4b20-b78f-cff131643350\") " pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"
Jul 31 02:27:17.295633 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.295551    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92de2058-3d2c-4b20-b78f-cff131643350-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9\" (UID: \"92de2058-3d2c-4b20-b78f-cff131643350\") " pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"
Jul 31 02:27:17.295633 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.295600    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7crb2\" (UniqueName: \"kubernetes.io/projected/cf4511df-478e-4cda-9b83-e6fac0490d3f-kube-api-access-7crb2\") pod \"payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n\" (UID: \"cf4511df-478e-4cda-9b83-e6fac0490d3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"
Jul 31 02:27:17.296303 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.296285    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92de2058-3d2c-4b20-b78f-cff131643350-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9\" (UID: \"92de2058-3d2c-4b20-b78f-cff131643350\") " pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"
Jul 31 02:27:17.303342 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.303322    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2drzd\" (UniqueName: \"kubernetes.io/projected/92de2058-3d2c-4b20-b78f-cff131643350-kube-api-access-2drzd\") pod \"payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9\" (UID: \"92de2058-3d2c-4b20-b78f-cff131643350\") " pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"
Jul 31 02:27:17.395262 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.395238    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"
Jul 31 02:27:17.396585 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.396542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4511df-478e-4cda-9b83-e6fac0490d3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n\" (UID: \"cf4511df-478e-4cda-9b83-e6fac0490d3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"
Jul 31 02:27:17.396691 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.396651    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7crb2\" (UniqueName: \"kubernetes.io/projected/cf4511df-478e-4cda-9b83-e6fac0490d3f-kube-api-access-7crb2\") pod \"payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n\" (UID: \"cf4511df-478e-4cda-9b83-e6fac0490d3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"
Jul 31 02:27:17.397245 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.397227    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4511df-478e-4cda-9b83-e6fac0490d3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n\" (UID: \"cf4511df-478e-4cda-9b83-e6fac0490d3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"
Jul 31 02:27:17.412944 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.412914    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc" event={"ID":"054b0c8e-37b7-4837-8b3d-5a6a422ed171","Type":"ContainerStarted","Data":"4e301479f08431bec62583bd0cf6b834e55d5ace4e08fac1c8f82329a93d4256"}
Jul 31 02:27:17.413041 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.412975    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc" event={"ID":"054b0c8e-37b7-4837-8b3d-5a6a422ed171","Type":"ContainerStarted","Data":"b4a5a7a7fa1a7ed792267ce1cc42879bdabb855a691d18e980356d891e078c16"}
Jul 31 02:27:17.413112 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.413098    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:17.423657 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.423637    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7crb2\" (UniqueName: \"kubernetes.io/projected/cf4511df-478e-4cda-9b83-e6fac0490d3f-kube-api-access-7crb2\") pod \"payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n\" (UID: \"cf4511df-478e-4cda-9b83-e6fac0490d3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"
Jul 31 02:27:17.436599 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.436379    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"
Jul 31 02:27:17.444678 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.443928    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc" podStartSLOduration=1.089786007 podStartE2EDuration="1.443909704s" totalImagesPullingTime="354.123697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:27:17.442069752 +0000 UTC m=+2995.807815391" watchObservedRunningTime="2026-07-31 02:27:17.443909704 +0000 UTC m=+2995.809655334"
Jul 31 02:27:17.537082 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.537053    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"]
Jul 31 02:27:17.540325 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:27:17.540298    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92de2058_3d2c_4b20_b78f_cff131643350.slice/crio-f0971026d28865825060d24fb958a4b463b83f0ac5c5d48260d548dcd564316d WatchSource:0}: Error finding container f0971026d28865825060d24fb958a4b463b83f0ac5c5d48260d548dcd564316d: Status 404 returned error can't find the container with id f0971026d28865825060d24fb958a4b463b83f0ac5c5d48260d548dcd564316d
Jul 31 02:27:17.797052 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:17.796980    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"]
Jul 31 02:27:17.801052 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:27:17.801025    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4511df_478e_4cda_9b83_e6fac0490d3f.slice/crio-3b5feeb2a88643b3ba45b1f81da3822b7eda83b943f65d226b7fc11bd6f59c47 WatchSource:0}: Error finding container 3b5feeb2a88643b3ba45b1f81da3822b7eda83b943f65d226b7fc11bd6f59c47: Status 404 returned error can't find the container with id 3b5feeb2a88643b3ba45b1f81da3822b7eda83b943f65d226b7fc11bd6f59c47
Jul 31 02:27:18.418625 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:18.418594    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n" event={"ID":"cf4511df-478e-4cda-9b83-e6fac0490d3f","Type":"ContainerStarted","Data":"88602f035ca93b90967a734a4b3f8a1fa296967fd6d174d34e58e9b5c3fba63e"}
Jul 31 02:27:18.418791 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:18.418650    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n" event={"ID":"cf4511df-478e-4cda-9b83-e6fac0490d3f","Type":"ContainerStarted","Data":"3b5feeb2a88643b3ba45b1f81da3822b7eda83b943f65d226b7fc11bd6f59c47"}
Jul 31 02:27:18.418791 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:18.418672    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"
Jul 31 02:27:18.420119 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:18.420088    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9" event={"ID":"92de2058-3d2c-4b20-b78f-cff131643350","Type":"ContainerStarted","Data":"8a81cbcec778f8b8c1eb65db5b8fef316dfc0adb3cf4d87ee9fc558c26d873a5"}
Jul 31 02:27:18.420249 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:18.420138    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9" event={"ID":"92de2058-3d2c-4b20-b78f-cff131643350","Type":"ContainerStarted","Data":"f0971026d28865825060d24fb958a4b463b83f0ac5c5d48260d548dcd564316d"}
Jul 31 02:27:18.434416 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:18.434370    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n" podStartSLOduration=1.434356492 podStartE2EDuration="1.434356492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:27:18.433241538 +0000 UTC m=+2996.798987165" watchObservedRunningTime="2026-07-31 02:27:18.434356492 +0000 UTC m=+2996.800102164"
Jul 31 02:27:18.448820 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:18.448783    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9" podStartSLOduration=1.448767895 podStartE2EDuration="1.448767895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:27:18.447185497 +0000 UTC m=+2996.812931137" watchObservedRunningTime="2026-07-31 02:27:18.448767895 +0000 UTC m=+2996.814513522"
Jul 31 02:27:19.424976 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:19.424945    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"
Jul 31 02:27:23.425733 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:23.425707    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:27:29.426683 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:29.426657    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"
Jul 31 02:27:30.431238 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:30.431205    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"
Jul 31 02:27:40.149570 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:40.149538    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:40.382265 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:40.382215    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:44.923960 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:44.923929    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:27:59.832700 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:27:59.832668    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:29:41.362051 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:29:41.362020    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:29:41.451369 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:29:41.451342    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:30:00.134014 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.133979    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-01a44c-29757750-jhs48"]
Jul 31 02:30:00.137521 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.137501    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757750-cmxtx"]
Jul 31 02:30:00.137674 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.137620    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-01a44c-29757750-jhs48"
Jul 31 02:30:00.144423 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.140379    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:30:00.145810 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.145790    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757750-cmxtx"
Jul 31 02:30:00.153609 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.153584    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp"]
Jul 31 02:30:00.153726 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.153697    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-01a44c-29757750-jhs48"
Jul 31 02:30:00.159287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.157243    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757750-cmxtx"]
Jul 31 02:30:00.159287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.157278    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-01a44c-29757750-jhs48"]
Jul 31 02:30:00.159287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.157299    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp"]
Jul 31 02:30:00.159287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.157371    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp"
Jul 31 02:30:00.163409 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.163392    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757750-cmxtx"
Jul 31 02:30:00.207230 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.207207    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp"
Jul 31 02:30:00.308425 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.308346    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-01a44c-29757750-jhs48"]
Jul 31 02:30:00.548885 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.548862    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757750-cmxtx"]
Jul 31 02:30:00.550902 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:30:00.550864    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb499d2_f7d7_44df_a0f0_db524cb64594.slice/crio-175f93759fc4cf435f07a8fd4faa049208fc601d2107d07273bf10b52a9b415c WatchSource:0}: Error finding container 175f93759fc4cf435f07a8fd4faa049208fc601d2107d07273bf10b52a9b415c: Status 404 returned error can't find the container with id 175f93759fc4cf435f07a8fd4faa049208fc601d2107d07273bf10b52a9b415c
Jul 31 02:30:00.577286 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:00.577267    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp"]
Jul 31 02:30:00.590455 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:30:00.590433    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fddcd1d_9836_4aee_8f30_d0b3d2220b60.slice/crio-6c34e81c9e2028206c3bf2558f8693bc0fc2f266f6ba84b0b68682d4c14bf994 WatchSource:0}: Error finding container 6c34e81c9e2028206c3bf2558f8693bc0fc2f266f6ba84b0b68682d4c14bf994: Status 404 returned error can't find the container with id 6c34e81c9e2028206c3bf2558f8693bc0fc2f266f6ba84b0b68682d4c14bf994
Jul 31 02:30:01.111006 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:01.110982    2556 generic.go:356] "Generic (PLEG): container finished" podID="ac87b666-bb42-4ca9-abf1-645bc1fd2b7c" containerID="7b2c0b1e7825b98593e6808254fbb9b026cb7662627205fa466e4755f152eb42" exitCode=0
Jul 31 02:30:01.111122 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:01.111068    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-01a44c-29757750-jhs48" event={"ID":"ac87b666-bb42-4ca9-abf1-645bc1fd2b7c","Type":"ContainerDied","Data":"7b2c0b1e7825b98593e6808254fbb9b026cb7662627205fa466e4755f152eb42"}
Jul 31 02:30:01.111180 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:01.111127    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-01a44c-29757750-jhs48" event={"ID":"ac87b666-bb42-4ca9-abf1-645bc1fd2b7c","Type":"ContainerStarted","Data":"cefbf9855bcb043f91d148536f2cc9680dc809f58f9aada5e79663419c45f6de"}
Jul 31 02:30:01.112442 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:01.112419    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757750-cmxtx" event={"ID":"7eb499d2-f7d7-44df-a0f0-db524cb64594","Type":"ContainerStarted","Data":"14c47d68592022796420889f5f3cb9388bf1914165599fd682a2ffa6d6d824b9"}
Jul 31 02:30:01.112584 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:01.112460    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757750-cmxtx" event={"ID":"7eb499d2-f7d7-44df-a0f0-db524cb64594","Type":"ContainerStarted","Data":"175f93759fc4cf435f07a8fd4faa049208fc601d2107d07273bf10b52a9b415c"}
Jul 31 02:30:01.113793 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:01.113767    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp" event={"ID":"7fddcd1d-9836-4aee-8f30-d0b3d2220b60","Type":"ContainerStarted","Data":"0233e38de75588bdb3acf1df1a99a0ccd41f6ffe15de6feac6992af2abdaaa41"}
Jul 31 02:30:01.113892 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:01.113813    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp" event={"ID":"7fddcd1d-9836-4aee-8f30-d0b3d2220b60","Type":"ContainerStarted","Data":"6c34e81c9e2028206c3bf2558f8693bc0fc2f266f6ba84b0b68682d4c14bf994"}
Jul 31 02:30:01.139409 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:01.139372    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757750-cmxtx" podStartSLOduration=1.139359981 podStartE2EDuration="1.139359981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:30:01.138185354 +0000 UTC m=+3159.503930981" watchObservedRunningTime="2026-07-31 02:30:01.139359981 +0000 UTC m=+3159.505105608"
Jul 31 02:30:01.152321 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:01.152282    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp" podStartSLOduration=1.152270398 podStartE2EDuration="1.152270398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:30:01.150720537 +0000 UTC m=+3159.516466175" watchObservedRunningTime="2026-07-31 02:30:01.152270398 +0000 UTC m=+3159.518016065"
Jul 31 02:30:02.119658 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:02.119628    2556 generic.go:356] "Generic (PLEG): container finished" podID="7eb499d2-f7d7-44df-a0f0-db524cb64594" containerID="14c47d68592022796420889f5f3cb9388bf1914165599fd682a2ffa6d6d824b9" exitCode=0
Jul 31 02:30:02.119773 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:02.119710    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757750-cmxtx" event={"ID":"7eb499d2-f7d7-44df-a0f0-db524cb64594","Type":"ContainerDied","Data":"14c47d68592022796420889f5f3cb9388bf1914165599fd682a2ffa6d6d824b9"}
Jul 31 02:30:02.121222 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:02.121202    2556 generic.go:356] "Generic (PLEG): container finished" podID="7fddcd1d-9836-4aee-8f30-d0b3d2220b60" containerID="0233e38de75588bdb3acf1df1a99a0ccd41f6ffe15de6feac6992af2abdaaa41" exitCode=0
Jul 31 02:30:02.121331 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:02.121294    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp" event={"ID":"7fddcd1d-9836-4aee-8f30-d0b3d2220b60","Type":"ContainerDied","Data":"0233e38de75588bdb3acf1df1a99a0ccd41f6ffe15de6feac6992af2abdaaa41"}
Jul 31 02:30:02.255973 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:02.255946    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-01a44c-29757750-jhs48"
Jul 31 02:30:03.126850 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:03.126828    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-01a44c-29757750-jhs48"
Jul 31 02:30:03.127008 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:03.126865    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-01a44c-29757750-jhs48" event={"ID":"ac87b666-bb42-4ca9-abf1-645bc1fd2b7c","Type":"ContainerDied","Data":"cefbf9855bcb043f91d148536f2cc9680dc809f58f9aada5e79663419c45f6de"}
Jul 31 02:30:03.127008 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:03.126904    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cefbf9855bcb043f91d148536f2cc9680dc809f58f9aada5e79663419c45f6de"
Jul 31 02:30:03.245550 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:03.245528    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp"
Jul 31 02:30:03.273878 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:03.273859    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757750-cmxtx"
Jul 31 02:30:04.132807 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:04.132780    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp"
Jul 31 02:30:04.132982 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:04.132815    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp" event={"ID":"7fddcd1d-9836-4aee-8f30-d0b3d2220b60","Type":"ContainerDied","Data":"6c34e81c9e2028206c3bf2558f8693bc0fc2f266f6ba84b0b68682d4c14bf994"}
Jul 31 02:30:04.132982 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:04.132895    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c34e81c9e2028206c3bf2558f8693bc0fc2f266f6ba84b0b68682d4c14bf994"
Jul 31 02:30:04.136736 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:04.134584    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757750-cmxtx" event={"ID":"7eb499d2-f7d7-44df-a0f0-db524cb64594","Type":"ContainerDied","Data":"175f93759fc4cf435f07a8fd4faa049208fc601d2107d07273bf10b52a9b415c"}
Jul 31 02:30:04.136736 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:04.134628    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="175f93759fc4cf435f07a8fd4faa049208fc601d2107d07273bf10b52a9b415c"
Jul 31 02:30:04.136736 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:04.134681    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757750-cmxtx"
Jul 31 02:30:22.969087 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:30:22.968993    2556 scope.go:117] "RemoveContainer" containerID="5587ce914b516a1a71173b4fb88dc83aea0428bca02c2b6df794520266b52386"
Jul 31 02:31:03.049968 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:31:03.049909    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:31:06.605776 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:31:06.605745    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:31:08.513812 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:31:08.513777    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:31:22.985914 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:31:22.985887    2556 scope.go:117] "RemoveContainer" containerID="f93204a7f3a31ad4374f548a7f546b3b95bc0119b0090f34f93fa4604b92bdea"
Jul 31 02:31:40.571576 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:31:40.571540    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:32:23.002413 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:32:23.002379    2556 scope.go:117] "RemoveContainer" containerID="681a04c5af02da7c7f03ab52b652c485b8c96cbf52c53505e515c6a8da442a32"
Jul 31 02:32:44.667862 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:32:44.667792    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:32:45.694863 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:32:45.694827    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:33:18.300867 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:33:18.297139    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:33:57.184601 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:33:57.184571    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:09.112591 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:09.112527    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:10.114237 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:10.114206    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:29.837176 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:29.837150    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:30.159601 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:30.159492    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:30.448813 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:30.448737    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:33.132960 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:33.132929    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:33.233181 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:33.233157    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:34.217807 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.217771    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"]
Jul 31 02:35:34.218299 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.218280    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb499d2-f7d7-44df-a0f0-db524cb64594" containerName="cleanup"
Jul 31 02:35:34.218342 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.218304    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb499d2-f7d7-44df-a0f0-db524cb64594" containerName="cleanup"
Jul 31 02:35:34.218342 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.218323    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac87b666-bb42-4ca9-abf1-645bc1fd2b7c" containerName="cleanup"
Jul 31 02:35:34.218342 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.218332    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac87b666-bb42-4ca9-abf1-645bc1fd2b7c" containerName="cleanup"
Jul 31 02:35:34.218428 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.218350    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fddcd1d-9836-4aee-8f30-d0b3d2220b60" containerName="cleanup"
Jul 31 02:35:34.218428 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.218359    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fddcd1d-9836-4aee-8f30-d0b3d2220b60" containerName="cleanup"
Jul 31 02:35:34.218490 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.218447    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac87b666-bb42-4ca9-abf1-645bc1fd2b7c" containerName="cleanup"
Jul 31 02:35:34.218490 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.218458    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb499d2-f7d7-44df-a0f0-db524cb64594" containerName="cleanup"
Jul 31 02:35:34.218490 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.218469    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fddcd1d-9836-4aee-8f30-d0b3d2220b60" containerName="cleanup"
Jul 31 02:35:34.221868 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.221846    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:34.223922 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.223902    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e0efa5-serving-cert\""
Jul 31 02:35:34.230053 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.229677    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"]
Jul 31 02:35:34.249758 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.249731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:34.338426 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.338406    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fae4c62-7dbc-4a61-9964-d815db304f6d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:34.338521 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.338460    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jft\" (UniqueName: \"kubernetes.io/projected/0fae4c62-7dbc-4a61-9964-d815db304f6d-kube-api-access-w7jft\") pod \"maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:34.338521 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.338505    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fae4c62-7dbc-4a61-9964-d815db304f6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:34.438780 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.438761    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jft\" (UniqueName: \"kubernetes.io/projected/0fae4c62-7dbc-4a61-9964-d815db304f6d-kube-api-access-w7jft\") pod \"maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:34.438871 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.438797    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fae4c62-7dbc-4a61-9964-d815db304f6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:34.438871 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.438826    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fae4c62-7dbc-4a61-9964-d815db304f6d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:34.438941 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:35:34.438910    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-serving-cert: secret "maas-api-e2e-ipp-a-e0efa5-serving-cert" not found
Jul 31 02:35:34.438983 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:35:34.438956    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fae4c62-7dbc-4a61-9964-d815db304f6d-maas-api-tls podName:0fae4c62-7dbc-4a61-9964-d815db304f6d nodeName:}" failed. No retries permitted until 2026-07-31 02:35:34.938941111 +0000 UTC m=+3493.304686716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0fae4c62-7dbc-4a61-9964-d815db304f6d-maas-api-tls") pod "maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn" (UID: "0fae4c62-7dbc-4a61-9964-d815db304f6d") : secret "maas-api-e2e-ipp-a-e0efa5-serving-cert" not found
Jul 31 02:35:34.439429 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.439409    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fae4c62-7dbc-4a61-9964-d815db304f6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:34.448928 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.448907    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jft\" (UniqueName: \"kubernetes.io/projected/0fae4c62-7dbc-4a61-9964-d815db304f6d-kube-api-access-w7jft\") pod \"maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:34.688048 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.688016    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:34.830118 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.830095    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:34.943807 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.943749    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fae4c62-7dbc-4a61-9964-d815db304f6d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:34.946196 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.946171    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fae4c62-7dbc-4a61-9964-d815db304f6d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:34.948279 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.948259    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:34.967744 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.967719    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"]
Jul 31 02:35:34.971083 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.971068    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"
Jul 31 02:35:34.973666 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.973650    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e0efa5\""
Jul 31 02:35:34.987358 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.987327    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"]
Jul 31 02:35:34.989522 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.989499    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"]
Jul 31 02:35:34.992887 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.992872    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"
Jul 31 02:35:34.998312 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:34.998293    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e0efa5-dockercfg-mnkkz\""
Jul 31 02:35:35.005729 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.005709    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"]
Jul 31 02:35:35.045123 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.045102    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b55133-3b5d-4124-921a-52eef9595441-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh\" (UID: \"f7b55133-3b5d-4124-921a-52eef9595441\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"
Jul 31 02:35:35.045220 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.045145    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg6lp\" (UniqueName: \"kubernetes.io/projected/f7b55133-3b5d-4124-921a-52eef9595441-kube-api-access-dg6lp\") pod \"payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh\" (UID: \"f7b55133-3b5d-4124-921a-52eef9595441\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"
Jul 31 02:35:35.133635 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.133606    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:35.145584 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.145543    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b55133-3b5d-4124-921a-52eef9595441-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh\" (UID: \"f7b55133-3b5d-4124-921a-52eef9595441\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"
Jul 31 02:35:35.145715 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.145601    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg6lp\" (UniqueName: \"kubernetes.io/projected/f7b55133-3b5d-4124-921a-52eef9595441-kube-api-access-dg6lp\") pod \"payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh\" (UID: \"f7b55133-3b5d-4124-921a-52eef9595441\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"
Jul 31 02:35:35.145715 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.145645    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f\" (UID: \"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"
Jul 31 02:35:35.145715 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.145705    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79dgw\" (UniqueName: \"kubernetes.io/projected/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7-kube-api-access-79dgw\") pod \"payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f\" (UID: \"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"
Jul 31 02:35:35.146217 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.146194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b55133-3b5d-4124-921a-52eef9595441-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh\" (UID: \"f7b55133-3b5d-4124-921a-52eef9595441\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"
Jul 31 02:35:35.153668 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.153645    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg6lp\" (UniqueName: \"kubernetes.io/projected/f7b55133-3b5d-4124-921a-52eef9595441-kube-api-access-dg6lp\") pod \"payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh\" (UID: \"f7b55133-3b5d-4124-921a-52eef9595441\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"
Jul 31 02:35:35.246298 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.246271    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f\" (UID: \"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"
Jul 31 02:35:35.246633 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.246344    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79dgw\" (UniqueName: \"kubernetes.io/projected/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7-kube-api-access-79dgw\") pod \"payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f\" (UID: \"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"
Jul 31 02:35:35.247301 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.247276    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f\" (UID: \"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"
Jul 31 02:35:35.257337 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.257310    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79dgw\" (UniqueName: \"kubernetes.io/projected/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7-kube-api-access-79dgw\") pod \"payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f\" (UID: \"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"
Jul 31 02:35:35.267801 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.267779    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"]
Jul 31 02:35:35.269815 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:35:35.269780    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fae4c62_7dbc_4a61_9964_d815db304f6d.slice/crio-6d26d60a21be8dd3ae2f220607a622b4137ccb59056287ee1dc85c9d732ded45 WatchSource:0}: Error finding container 6d26d60a21be8dd3ae2f220607a622b4137ccb59056287ee1dc85c9d732ded45: Status 404 returned error can't find the container with id 6d26d60a21be8dd3ae2f220607a622b4137ccb59056287ee1dc85c9d732ded45
Jul 31 02:35:35.271376 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.271361    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 02:35:35.281392 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.281369    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"
Jul 31 02:35:35.302198 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.302178    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"
Jul 31 02:35:35.427714 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.427670    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"]
Jul 31 02:35:35.429692 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:35:35.429652    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b55133_3b5d_4124_921a_52eef9595441.slice/crio-badb3919ac4227107ed9171c89900fff8e989c4a0cbf0159f469b4e308afdcc9 WatchSource:0}: Error finding container badb3919ac4227107ed9171c89900fff8e989c4a0cbf0159f469b4e308afdcc9: Status 404 returned error can't find the container with id badb3919ac4227107ed9171c89900fff8e989c4a0cbf0159f469b4e308afdcc9
Jul 31 02:35:35.449064 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.449040    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"]
Jul 31 02:35:35.450989 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:35:35.450953    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ff9f8f_526d_4445_b3d4_9ccf64c82cb7.slice/crio-bb712e34d3ee1fc2a905beaf3fd376856b6223c76c7800c69313cd1453df835e WatchSource:0}: Error finding container bb712e34d3ee1fc2a905beaf3fd376856b6223c76c7800c69313cd1453df835e: Status 404 returned error can't find the container with id bb712e34d3ee1fc2a905beaf3fd376856b6223c76c7800c69313cd1453df835e
Jul 31 02:35:35.550641 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.550597    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh" event={"ID":"f7b55133-3b5d-4124-921a-52eef9595441","Type":"ContainerStarted","Data":"ad4773253748507217a3194b896eb1a2451715947f0207a8f1e964a456da9e50"}
Jul 31 02:35:35.550791 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.550670    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh" event={"ID":"f7b55133-3b5d-4124-921a-52eef9595441","Type":"ContainerStarted","Data":"badb3919ac4227107ed9171c89900fff8e989c4a0cbf0159f469b4e308afdcc9"}
Jul 31 02:35:35.550863 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.550803    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"
Jul 31 02:35:35.551772 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.551741    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn" event={"ID":"0fae4c62-7dbc-4a61-9964-d815db304f6d","Type":"ContainerStarted","Data":"6d26d60a21be8dd3ae2f220607a622b4137ccb59056287ee1dc85c9d732ded45"}
Jul 31 02:35:35.553160 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.553138    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f" event={"ID":"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7","Type":"ContainerStarted","Data":"54e2ff68100f49f2d81372fda03a8d95ebeebc8a908aaf1172142597b1033845"}
Jul 31 02:35:35.553252 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.553176    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f" event={"ID":"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7","Type":"ContainerStarted","Data":"bb712e34d3ee1fc2a905beaf3fd376856b6223c76c7800c69313cd1453df835e"}
Jul 31 02:35:35.553252 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.553216    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"
Jul 31 02:35:35.564879 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.564837    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh" podStartSLOduration=1.5648250049999999 podStartE2EDuration="1.564825005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:35:35.563819242 +0000 UTC m=+3493.929564881" watchObservedRunningTime="2026-07-31 02:35:35.564825005 +0000 UTC m=+3493.930570630"
Jul 31 02:35:35.579977 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:35.579931    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f" podStartSLOduration=1.5799149689999998 podStartE2EDuration="1.579914969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:35:35.578053511 +0000 UTC m=+3493.943799137" watchObservedRunningTime="2026-07-31 02:35:35.579914969 +0000 UTC m=+3493.945660595"
Jul 31 02:35:36.559997 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:36.559965    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn" event={"ID":"0fae4c62-7dbc-4a61-9964-d815db304f6d","Type":"ContainerStarted","Data":"a3f1f2b62efa95729293e9ca151570df7aa6c93c169cc5ccee291fdb6445086f"}
Jul 31 02:35:36.560375 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:36.560021    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:36.575471 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:36.575420    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn" podStartSLOduration=2.077114837 podStartE2EDuration="2.575400144s" totalImagesPullingTime="498.285307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:35:36.57404149 +0000 UTC m=+3494.939787118" watchObservedRunningTime="2026-07-31 02:35:36.575400144 +0000 UTC m=+3494.941145775"
Jul 31 02:35:42.569993 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:42.569969    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:35:46.562065 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:46.562037    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"
Jul 31 02:35:46.562416 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:46.562156    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"
Jul 31 02:35:58.248131 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:58.248097    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:58.930005 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:58.929976    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:59.239361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:59.239283    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:35:59.343036 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:35:59.343008    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:36:02.382098 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:02.382063    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:36:02.534545 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:02.534500    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:36:03.444023 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.443984    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"]
Jul 31 02:36:03.448699 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.448683    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:03.450718 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.450698    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3dc29c-serving-cert\""
Jul 31 02:36:03.455780 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.455728    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"]
Jul 31 02:36:03.473000 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.472978    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:36:03.562054 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.562029    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9v28\" (UniqueName: \"kubernetes.io/projected/b910e789-7d08-4378-89b1-b9e8f9aef317-kube-api-access-s9v28\") pod \"maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl\" (UID: \"b910e789-7d08-4378-89b1-b9e8f9aef317\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:03.562161 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.562071    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b910e789-7d08-4378-89b1-b9e8f9aef317-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl\" (UID: \"b910e789-7d08-4378-89b1-b9e8f9aef317\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:03.562161 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.562096    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b910e789-7d08-4378-89b1-b9e8f9aef317-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl\" (UID: \"b910e789-7d08-4378-89b1-b9e8f9aef317\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:03.662803 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.662778    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9v28\" (UniqueName: \"kubernetes.io/projected/b910e789-7d08-4378-89b1-b9e8f9aef317-kube-api-access-s9v28\") pod \"maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl\" (UID: \"b910e789-7d08-4378-89b1-b9e8f9aef317\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:03.662913 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.662821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b910e789-7d08-4378-89b1-b9e8f9aef317-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl\" (UID: \"b910e789-7d08-4378-89b1-b9e8f9aef317\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:03.662913 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.662844    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b910e789-7d08-4378-89b1-b9e8f9aef317-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl\" (UID: \"b910e789-7d08-4378-89b1-b9e8f9aef317\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:03.663424 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.663393    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b910e789-7d08-4378-89b1-b9e8f9aef317-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl\" (UID: \"b910e789-7d08-4378-89b1-b9e8f9aef317\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:03.665134 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.665116    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b910e789-7d08-4378-89b1-b9e8f9aef317-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl\" (UID: \"b910e789-7d08-4378-89b1-b9e8f9aef317\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:03.670032 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.670011    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9v28\" (UniqueName: \"kubernetes.io/projected/b910e789-7d08-4378-89b1-b9e8f9aef317-kube-api-access-s9v28\") pod \"maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl\" (UID: \"b910e789-7d08-4378-89b1-b9e8f9aef317\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:03.761319 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.761271    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:03.887135 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.887111    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"]
Jul 31 02:36:03.890317 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:36:03.890286    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb910e789_7d08_4378_89b1_b9e8f9aef317.slice/crio-22ef18f1867529b95b39f4e594eaf4852f2e57776d1cd6eae5ed4e7def322a48 WatchSource:0}: Error finding container 22ef18f1867529b95b39f4e594eaf4852f2e57776d1cd6eae5ed4e7def322a48: Status 404 returned error can't find the container with id 22ef18f1867529b95b39f4e594eaf4852f2e57776d1cd6eae5ed4e7def322a48
Jul 31 02:36:03.922387 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:03.922363    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:36:04.021143 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.021080    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:36:04.054098 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.054073    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"]
Jul 31 02:36:04.058846 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.058832    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"
Jul 31 02:36:04.062803 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.062777    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3dc29c\""
Jul 31 02:36:04.062888 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.062828    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3dc29c-dockercfg-bkwdc\""
Jul 31 02:36:04.073144 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.073123    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"]
Jul 31 02:36:04.096064 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.096036    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"]
Jul 31 02:36:04.099218 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.099199    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:36:04.099414 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.099400    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"
Jul 31 02:36:04.109330 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.109309    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"]
Jul 31 02:36:04.165808 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.165783    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2kz\" (UniqueName: \"kubernetes.io/projected/5d388446-1b53-4e27-ba67-6afa0eefbd9e-kube-api-access-6m2kz\") pod \"payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv\" (UID: \"5d388446-1b53-4e27-ba67-6afa0eefbd9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"
Jul 31 02:36:04.165956 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.165849    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d388446-1b53-4e27-ba67-6afa0eefbd9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv\" (UID: \"5d388446-1b53-4e27-ba67-6afa0eefbd9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"
Jul 31 02:36:04.267031 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.266995    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2kz\" (UniqueName: \"kubernetes.io/projected/5d388446-1b53-4e27-ba67-6afa0eefbd9e-kube-api-access-6m2kz\") pod \"payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv\" (UID: \"5d388446-1b53-4e27-ba67-6afa0eefbd9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"
Jul 31 02:36:04.267157 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.267107    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dffbfa0-415d-4c07-bd36-ca787db74c57-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w\" (UID: \"0dffbfa0-415d-4c07-bd36-ca787db74c57\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"
Jul 31 02:36:04.267203 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.267155    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d388446-1b53-4e27-ba67-6afa0eefbd9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv\" (UID: \"5d388446-1b53-4e27-ba67-6afa0eefbd9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"
Jul 31 02:36:04.267249 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.267212    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t2rw\" (UniqueName: \"kubernetes.io/projected/0dffbfa0-415d-4c07-bd36-ca787db74c57-kube-api-access-2t2rw\") pod \"payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w\" (UID: \"0dffbfa0-415d-4c07-bd36-ca787db74c57\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"
Jul 31 02:36:04.267967 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.267949    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d388446-1b53-4e27-ba67-6afa0eefbd9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv\" (UID: \"5d388446-1b53-4e27-ba67-6afa0eefbd9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"
Jul 31 02:36:04.274543 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.274501    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2kz\" (UniqueName: \"kubernetes.io/projected/5d388446-1b53-4e27-ba67-6afa0eefbd9e-kube-api-access-6m2kz\") pod \"payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv\" (UID: \"5d388446-1b53-4e27-ba67-6afa0eefbd9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"
Jul 31 02:36:04.368524 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.368502    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"
Jul 31 02:36:04.368802 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.368779    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dffbfa0-415d-4c07-bd36-ca787db74c57-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w\" (UID: \"0dffbfa0-415d-4c07-bd36-ca787db74c57\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"
Jul 31 02:36:04.368904 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.368823    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2rw\" (UniqueName: \"kubernetes.io/projected/0dffbfa0-415d-4c07-bd36-ca787db74c57-kube-api-access-2t2rw\") pod \"payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w\" (UID: \"0dffbfa0-415d-4c07-bd36-ca787db74c57\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"
Jul 31 02:36:04.369536 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.369514    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dffbfa0-415d-4c07-bd36-ca787db74c57-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w\" (UID: \"0dffbfa0-415d-4c07-bd36-ca787db74c57\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"
Jul 31 02:36:04.376292 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.376273    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t2rw\" (UniqueName: \"kubernetes.io/projected/0dffbfa0-415d-4c07-bd36-ca787db74c57-kube-api-access-2t2rw\") pod \"payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w\" (UID: \"0dffbfa0-415d-4c07-bd36-ca787db74c57\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"
Jul 31 02:36:04.409147 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.409115    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"
Jul 31 02:36:04.510446 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.510385    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"]
Jul 31 02:36:04.513164 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:36:04.513138    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d388446_1b53_4e27_ba67_6afa0eefbd9e.slice/crio-9909ae7f745d909128d8fc0246f3dc5d583fad688bd42bc3b54d35553e9b3ec4 WatchSource:0}: Error finding container 9909ae7f745d909128d8fc0246f3dc5d583fad688bd42bc3b54d35553e9b3ec4: Status 404 returned error can't find the container with id 9909ae7f745d909128d8fc0246f3dc5d583fad688bd42bc3b54d35553e9b3ec4
Jul 31 02:36:04.563593 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.563551    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"]
Jul 31 02:36:04.564874 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:36:04.564850    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dffbfa0_415d_4c07_bd36_ca787db74c57.slice/crio-6bce193fa3eaaa6308abe0e7a6a8b3a3c2472d14db6620ce924094c21c1c4b13 WatchSource:0}: Error finding container 6bce193fa3eaaa6308abe0e7a6a8b3a3c2472d14db6620ce924094c21c1c4b13: Status 404 returned error can't find the container with id 6bce193fa3eaaa6308abe0e7a6a8b3a3c2472d14db6620ce924094c21c1c4b13
Jul 31 02:36:04.688048 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.688012    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv" event={"ID":"5d388446-1b53-4e27-ba67-6afa0eefbd9e","Type":"ContainerStarted","Data":"c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798"}
Jul 31 02:36:04.688213 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.688071    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv" event={"ID":"5d388446-1b53-4e27-ba67-6afa0eefbd9e","Type":"ContainerStarted","Data":"9909ae7f745d909128d8fc0246f3dc5d583fad688bd42bc3b54d35553e9b3ec4"}
Jul 31 02:36:04.688213 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.688131    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"
Jul 31 02:36:04.689894 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.689871    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w" event={"ID":"0dffbfa0-415d-4c07-bd36-ca787db74c57","Type":"ContainerStarted","Data":"e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790"}
Jul 31 02:36:04.690012 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.689911    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w" event={"ID":"0dffbfa0-415d-4c07-bd36-ca787db74c57","Type":"ContainerStarted","Data":"6bce193fa3eaaa6308abe0e7a6a8b3a3c2472d14db6620ce924094c21c1c4b13"}
Jul 31 02:36:04.690012 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.689974    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"
Jul 31 02:36:04.691484 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.691462    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl" event={"ID":"b910e789-7d08-4378-89b1-b9e8f9aef317","Type":"ContainerStarted","Data":"487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4"}
Jul 31 02:36:04.691612 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.691504    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl" event={"ID":"b910e789-7d08-4378-89b1-b9e8f9aef317","Type":"ContainerStarted","Data":"22ef18f1867529b95b39f4e594eaf4852f2e57776d1cd6eae5ed4e7def322a48"}
Jul 31 02:36:04.691612 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.691537    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:04.703863 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.703816    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv" podStartSLOduration=0.703800264 podStartE2EDuration="703.800264ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:36:04.702364658 +0000 UTC m=+3523.068110742" watchObservedRunningTime="2026-07-31 02:36:04.703800264 +0000 UTC m=+3523.069545889"
Jul 31 02:36:04.717893 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.717857    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl" podStartSLOduration=1.333483134 podStartE2EDuration="1.717845168s" totalImagesPullingTime="384.362034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:36:04.71683904 +0000 UTC m=+3523.082584662" watchObservedRunningTime="2026-07-31 02:36:04.717845168 +0000 UTC m=+3523.083590808"
Jul 31 02:36:04.732115 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:04.732079    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w" podStartSLOduration=0.73206699 podStartE2EDuration="732.06699ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:36:04.729908764 +0000 UTC m=+3523.095654390" watchObservedRunningTime="2026-07-31 02:36:04.73206699 +0000 UTC m=+3523.097812617"
Jul 31 02:36:10.398584 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:10.398538    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:36:10.701236 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:10.701177    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:36:15.698959 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:15.698931    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"
Jul 31 02:36:15.699357 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:15.699111    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"
Jul 31 02:36:27.431682 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:27.431651    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:36:34.435888 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:34.435857    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-l2mkg_fa01e1a2-7436-45ba-b058-346e7534e8ec/payload-processing/0.log"
Jul 31 02:36:34.558802 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:34.558778    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w_0dffbfa0-415d-4c07-bd36-ca787db74c57/payload-processing/0.log"
Jul 31 02:36:39.414772 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:36:39.414742    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:15.363226 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:15.363156    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:15.420895 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:15.420872    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:29.606020 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:29.605989    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f_c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7/payload-processing/0.log"
Jul 31 02:37:29.728583 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:29.728545    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-l2mkg_fa01e1a2-7436-45ba-b058-346e7534e8ec/payload-processing/0.log"
Jul 31 02:37:29.860886 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:29.860830    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w_0dffbfa0-415d-4c07-bd36-ca787db74c57/payload-processing/0.log"
Jul 31 02:37:30.144331 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:30.144279    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:30.457194 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:30.457121    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:30.588898 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:30.588870    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:33.830506 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:33.830482    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:33.926701 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:33.926674    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:34.916756 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:34.916728    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"]
Jul 31 02:37:34.920414 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:34.920381    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:34.922493 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:34.922466    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-9daafd-serving-cert\""
Jul 31 02:37:34.930428 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:34.930409    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"]
Jul 31 02:37:34.951275 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:34.951250    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:35.038920 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.038894    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:35.039029 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.038960    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:35.039029 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.039004    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4sb\" (UniqueName: \"kubernetes.io/projected/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-kube-api-access-ks4sb\") pod \"maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:35.139680 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.139655    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:35.139796 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.139705    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:35.139796 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.139726    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4sb\" (UniqueName: \"kubernetes.io/projected/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-kube-api-access-ks4sb\") pod \"maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:35.139916 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:37:35.139809    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-serving-cert: secret "maas-api-e2e-ipp-cleanup-9daafd-serving-cert" not found
Jul 31 02:37:35.139916 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:37:35.139881    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-maas-api-tls podName:5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8 nodeName:}" failed. No retries permitted until 2026-07-31 02:37:35.639861971 +0000 UTC m=+3614.005607576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr" (UID: "5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8") : secret "maas-api-e2e-ipp-cleanup-9daafd-serving-cert" not found
Jul 31 02:37:35.140303 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.140280    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:35.147688 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.147670    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4sb\" (UniqueName: \"kubernetes.io/projected/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-kube-api-access-ks4sb\") pod \"maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:35.207343 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.207288    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:35.441817 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.441796    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:35.538795 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.538718    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:35.566490 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.566467    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"]
Jul 31 02:37:35.570060 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.570045    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"
Jul 31 02:37:35.572137 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.572117    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-9daafd\""
Jul 31 02:37:35.572243 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.572147    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-9daafd-dockercfg-hfkwj\""
Jul 31 02:37:35.579884 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.579866    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"]
Jul 31 02:37:35.642518 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.642496    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:35.644978 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.644951    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:35.742863 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.742835    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7dkr\" (UniqueName: \"kubernetes.io/projected/39ecabc1-4e4b-4d53-b805-6c46c5890606-kube-api-access-v7dkr\") pod \"payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q\" (UID: \"39ecabc1-4e4b-4d53-b805-6c46c5890606\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"
Jul 31 02:37:35.742998 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.742881    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ecabc1-4e4b-4d53-b805-6c46c5890606-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q\" (UID: \"39ecabc1-4e4b-4d53-b805-6c46c5890606\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"
Jul 31 02:37:35.831659 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.831634    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:35.843727 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.843707    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ecabc1-4e4b-4d53-b805-6c46c5890606-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q\" (UID: \"39ecabc1-4e4b-4d53-b805-6c46c5890606\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"
Jul 31 02:37:35.843800 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.843783    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7dkr\" (UniqueName: \"kubernetes.io/projected/39ecabc1-4e4b-4d53-b805-6c46c5890606-kube-api-access-v7dkr\") pod \"payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q\" (UID: \"39ecabc1-4e4b-4d53-b805-6c46c5890606\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"
Jul 31 02:37:35.844527 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.844506    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ecabc1-4e4b-4d53-b805-6c46c5890606-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q\" (UID: \"39ecabc1-4e4b-4d53-b805-6c46c5890606\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"
Jul 31 02:37:35.854361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.854342    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7dkr\" (UniqueName: \"kubernetes.io/projected/39ecabc1-4e4b-4d53-b805-6c46c5890606-kube-api-access-v7dkr\") pod \"payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q\" (UID: \"39ecabc1-4e4b-4d53-b805-6c46c5890606\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"
Jul 31 02:37:35.882376 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.882285    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"
Jul 31 02:37:35.975265 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:35.975178    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"]
Jul 31 02:37:35.978089 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:37:35.978033    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd8cff8_2dd4_4aaa_bc52_e6c5491547b8.slice/crio-4687fb02f84ba1ef12e87a187b3014161ccd94cb91908c25423faba50c40bef1 WatchSource:0}: Error finding container 4687fb02f84ba1ef12e87a187b3014161ccd94cb91908c25423faba50c40bef1: Status 404 returned error can't find the container with id 4687fb02f84ba1ef12e87a187b3014161ccd94cb91908c25423faba50c40bef1
Jul 31 02:37:36.024816 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:36.024783    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"]
Jul 31 02:37:36.026332 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:37:36.026308    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ecabc1_4e4b_4d53_b805_6c46c5890606.slice/crio-6f89011e5666e5ce03f35d00304f753a3f65458227b0fbc50278d7b598950abc WatchSource:0}: Error finding container 6f89011e5666e5ce03f35d00304f753a3f65458227b0fbc50278d7b598950abc: Status 404 returned error can't find the container with id 6f89011e5666e5ce03f35d00304f753a3f65458227b0fbc50278d7b598950abc
Jul 31 02:37:36.123729 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:36.123660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr" event={"ID":"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8","Type":"ContainerStarted","Data":"4687fb02f84ba1ef12e87a187b3014161ccd94cb91908c25423faba50c40bef1"}
Jul 31 02:37:36.125159 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:36.125133    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q" event={"ID":"39ecabc1-4e4b-4d53-b805-6c46c5890606","Type":"ContainerStarted","Data":"014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554"}
Jul 31 02:37:36.125279 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:36.125180    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q" event={"ID":"39ecabc1-4e4b-4d53-b805-6c46c5890606","Type":"ContainerStarted","Data":"6f89011e5666e5ce03f35d00304f753a3f65458227b0fbc50278d7b598950abc"}
Jul 31 02:37:36.125279 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:36.125236    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"
Jul 31 02:37:36.140840 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:36.140802    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q" podStartSLOduration=1.140790747 podStartE2EDuration="1.140790747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:37:36.140036498 +0000 UTC m=+3614.505782124" watchObservedRunningTime="2026-07-31 02:37:36.140790747 +0000 UTC m=+3614.506536407"
Jul 31 02:37:37.131451 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:37.131414    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr" event={"ID":"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8","Type":"ContainerStarted","Data":"2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8"}
Jul 31 02:37:37.131825 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:37.131717    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:37.147335 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:37.147286    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr" podStartSLOduration=2.621895655 podStartE2EDuration="3.147272336s" totalImagesPullingTime="525.376681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 02:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:37:37.145663942 +0000 UTC m=+3615.511409568" watchObservedRunningTime="2026-07-31 02:37:37.147272336 +0000 UTC m=+3615.513017961"
Jul 31 02:37:43.140906 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:43.140874    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:37:46.787356 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:46.787321    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:47.045164 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:47.045107    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:37:47.134213 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:47.134189    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"
Jul 31 02:37:58.922970 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:37:58.922925    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:15.418004 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.417964    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"]
Jul 31 02:38:15.418502 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.418288    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr" podUID="5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8" containerName="maas-api" containerID="cri-o://2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8" gracePeriod=30
Jul 31 02:38:15.460612 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.460551    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:15.473708 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.473682    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"]
Jul 31 02:38:15.473886 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.473860    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q" podUID="39ecabc1-4e4b-4d53-b805-6c46c5890606" containerName="payload-processing" containerID="cri-o://014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554" gracePeriod=30
Jul 31 02:38:15.557985 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.557959    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:15.618541 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.618510    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:15.723658 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.723631    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:38:15.757657 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.757635    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"
Jul 31 02:38:15.850649 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.850623    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-openshift-service-ca\") pod \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") "
Jul 31 02:38:15.850767 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.850666    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7dkr\" (UniqueName: \"kubernetes.io/projected/39ecabc1-4e4b-4d53-b805-6c46c5890606-kube-api-access-v7dkr\") pod \"39ecabc1-4e4b-4d53-b805-6c46c5890606\" (UID: \"39ecabc1-4e4b-4d53-b805-6c46c5890606\") "
Jul 31 02:38:15.850767 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.850682    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ecabc1-4e4b-4d53-b805-6c46c5890606-plugins-config-volume\") pod \"39ecabc1-4e4b-4d53-b805-6c46c5890606\" (UID: \"39ecabc1-4e4b-4d53-b805-6c46c5890606\") "
Jul 31 02:38:15.850877 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.850813    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-maas-api-tls\") pod \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") "
Jul 31 02:38:15.850877 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.850854    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks4sb\" (UniqueName: \"kubernetes.io/projected/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-kube-api-access-ks4sb\") pod \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\" (UID: \"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8\") "
Jul 31 02:38:15.850988 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.850960    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8" (UID: "5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:38:15.851053 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.851029    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ecabc1-4e4b-4d53-b805-6c46c5890606-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39ecabc1-4e4b-4d53-b805-6c46c5890606" (UID: "39ecabc1-4e4b-4d53-b805-6c46c5890606"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:38:15.851189 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.851172    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:38:15.851268 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.851190    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ecabc1-4e4b-4d53-b805-6c46c5890606-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:38:15.852771 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.852744    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ecabc1-4e4b-4d53-b805-6c46c5890606-kube-api-access-v7dkr" (OuterVolumeSpecName: "kube-api-access-v7dkr") pod "39ecabc1-4e4b-4d53-b805-6c46c5890606" (UID: "39ecabc1-4e4b-4d53-b805-6c46c5890606"). InnerVolumeSpecName "kube-api-access-v7dkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:38:15.852877 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.852858    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-kube-api-access-ks4sb" (OuterVolumeSpecName: "kube-api-access-ks4sb") pod "5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8" (UID: "5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8"). InnerVolumeSpecName "kube-api-access-ks4sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:38:15.852922 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.852862    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8" (UID: "5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:38:15.951530 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.951482    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7dkr\" (UniqueName: \"kubernetes.io/projected/39ecabc1-4e4b-4d53-b805-6c46c5890606-kube-api-access-v7dkr\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:38:15.951530 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.951501    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:38:15.951530 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:15.951510    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks4sb\" (UniqueName: \"kubernetes.io/projected/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8-kube-api-access-ks4sb\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:38:16.038376 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.038352    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:16.312127 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.312102    2556 generic.go:356] "Generic (PLEG): container finished" podID="5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8" containerID="2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8" exitCode=0
Jul 31 02:38:16.312217 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.312144    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"
Jul 31 02:38:16.312217 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.312183    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr" event={"ID":"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8","Type":"ContainerDied","Data":"2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8"}
Jul 31 02:38:16.312339 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.312231    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr" event={"ID":"5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8","Type":"ContainerDied","Data":"4687fb02f84ba1ef12e87a187b3014161ccd94cb91908c25423faba50c40bef1"}
Jul 31 02:38:16.312339 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.312259    2556 scope.go:117] "RemoveContainer" containerID="2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8"
Jul 31 02:38:16.313602 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.313579    2556 generic.go:356] "Generic (PLEG): container finished" podID="39ecabc1-4e4b-4d53-b805-6c46c5890606" containerID="014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554" exitCode=0
Jul 31 02:38:16.313695 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.313615    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"
Jul 31 02:38:16.313695 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.313635    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q" event={"ID":"39ecabc1-4e4b-4d53-b805-6c46c5890606","Type":"ContainerDied","Data":"014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554"}
Jul 31 02:38:16.313695 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.313674    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q" event={"ID":"39ecabc1-4e4b-4d53-b805-6c46c5890606","Type":"ContainerDied","Data":"6f89011e5666e5ce03f35d00304f753a3f65458227b0fbc50278d7b598950abc"}
Jul 31 02:38:16.320552 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.320537    2556 scope.go:117] "RemoveContainer" containerID="2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8"
Jul 31 02:38:16.320827 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:38:16.320809    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8\": container with ID starting with 2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8 not found: ID does not exist" containerID="2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8"
Jul 31 02:38:16.320899 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.320838    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8"} err="failed to get container status \"2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8\": rpc error: code = NotFound desc = could not find container \"2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8\": container with ID starting with 2fd06365d6f04a584cab40b7e2db408e1abbd23b3bb9548330e542888fbb0cc8 not found: ID does not exist"
Jul 31 02:38:16.320899 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.320860    2556 scope.go:117] "RemoveContainer" containerID="014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554"
Jul 31 02:38:16.328371 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.328358    2556 scope.go:117] "RemoveContainer" containerID="014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554"
Jul 31 02:38:16.328612 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:38:16.328595    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554\": container with ID starting with 014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554 not found: ID does not exist" containerID="014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554"
Jul 31 02:38:16.328705 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.328619    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554"} err="failed to get container status \"014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554\": rpc error: code = NotFound desc = could not find container \"014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554\": container with ID starting with 014f4ee22c5db64108bb4846847b2cc02a1015798e9f2ca7a20673d11c3fd554 not found: ID does not exist"
Jul 31 02:38:16.329634 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.329617    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"]
Jul 31 02:38:16.333283 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.333264    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9daafd-66878878b-ncskr"]
Jul 31 02:38:16.341987 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.341965    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"]
Jul 31 02:38:16.345844 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:16.345827    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9daafd-775486d4b6-4r54q"]
Jul 31 02:38:18.152041 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:18.152006    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39ecabc1-4e4b-4d53-b805-6c46c5890606" path="/var/lib/kubelet/pods/39ecabc1-4e4b-4d53-b805-6c46c5890606/volumes"
Jul 31 02:38:18.152713 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:18.152692    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8" path="/var/lib/kubelet/pods/5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8/volumes"
Jul 31 02:38:21.730768 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:21.730738    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:21.831359 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:21.831332    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:21.966620 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:21.966590    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:22.229242 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:22.229215    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:41.163354 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.163317    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"]
Jul 31 02:38:41.163831 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.163659    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn" podUID="0fae4c62-7dbc-4a61-9964-d815db304f6d" containerName="maas-api" containerID="cri-o://a3f1f2b62efa95729293e9ca151570df7aa6c93c169cc5ccee291fdb6445086f" gracePeriod=30
Jul 31 02:38:41.189917 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.189896    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:41.215526 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.215494    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"]
Jul 31 02:38:41.215820 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.215790    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f" podUID="c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7" containerName="payload-processing" containerID="cri-o://54e2ff68100f49f2d81372fda03a8d95ebeebc8a908aaf1172142597b1033845" gracePeriod=30
Jul 31 02:38:41.224764 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.224740    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"]
Jul 31 02:38:41.224992 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.224969    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh" podUID="f7b55133-3b5d-4124-921a-52eef9595441" containerName="payload-pre-processing" containerID="cri-o://ad4773253748507217a3194b896eb1a2451715947f0207a8f1e964a456da9e50" gracePeriod=30
Jul 31 02:38:41.258097 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.258075    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:41.329546 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.329517    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:41.391426 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.391396    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:41.429170 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.429095    2556 generic.go:356] "Generic (PLEG): container finished" podID="f7b55133-3b5d-4124-921a-52eef9595441" containerID="ad4773253748507217a3194b896eb1a2451715947f0207a8f1e964a456da9e50" exitCode=0
Jul 31 02:38:41.429314 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.429164    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh" event={"ID":"f7b55133-3b5d-4124-921a-52eef9595441","Type":"ContainerDied","Data":"ad4773253748507217a3194b896eb1a2451715947f0207a8f1e964a456da9e50"}
Jul 31 02:38:41.430687 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.430661    2556 generic.go:356] "Generic (PLEG): container finished" podID="0fae4c62-7dbc-4a61-9964-d815db304f6d" containerID="a3f1f2b62efa95729293e9ca151570df7aa6c93c169cc5ccee291fdb6445086f" exitCode=0
Jul 31 02:38:41.430807 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.430734    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn" event={"ID":"0fae4c62-7dbc-4a61-9964-d815db304f6d","Type":"ContainerDied","Data":"a3f1f2b62efa95729293e9ca151570df7aa6c93c169cc5ccee291fdb6445086f"}
Jul 31 02:38:41.432133 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.432114    2556 generic.go:356] "Generic (PLEG): container finished" podID="c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7" containerID="54e2ff68100f49f2d81372fda03a8d95ebeebc8a908aaf1172142597b1033845" exitCode=0
Jul 31 02:38:41.432224 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.432191    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f" event={"ID":"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7","Type":"ContainerDied","Data":"54e2ff68100f49f2d81372fda03a8d95ebeebc8a908aaf1172142597b1033845"}
Jul 31 02:38:41.479148 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.479126    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"
Jul 31 02:38:41.515352 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.515326    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:38:41.527674 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.527655    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"
Jul 31 02:38:41.655026 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.655000    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b55133-3b5d-4124-921a-52eef9595441-plugins-config-volume\") pod \"f7b55133-3b5d-4124-921a-52eef9595441\" (UID: \"f7b55133-3b5d-4124-921a-52eef9595441\") "
Jul 31 02:38:41.655151 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.655046    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79dgw\" (UniqueName: \"kubernetes.io/projected/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7-kube-api-access-79dgw\") pod \"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7\" (UID: \"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7\") "
Jul 31 02:38:41.655151 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.655070    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fae4c62-7dbc-4a61-9964-d815db304f6d-openshift-service-ca\") pod \"0fae4c62-7dbc-4a61-9964-d815db304f6d\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") "
Jul 31 02:38:41.655151 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.655087    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg6lp\" (UniqueName: \"kubernetes.io/projected/f7b55133-3b5d-4124-921a-52eef9595441-kube-api-access-dg6lp\") pod \"f7b55133-3b5d-4124-921a-52eef9595441\" (UID: \"f7b55133-3b5d-4124-921a-52eef9595441\") "
Jul 31 02:38:41.655151 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.655113    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7jft\" (UniqueName: \"kubernetes.io/projected/0fae4c62-7dbc-4a61-9964-d815db304f6d-kube-api-access-w7jft\") pod \"0fae4c62-7dbc-4a61-9964-d815db304f6d\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") "
Jul 31 02:38:41.655346 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.655169    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fae4c62-7dbc-4a61-9964-d815db304f6d-maas-api-tls\") pod \"0fae4c62-7dbc-4a61-9964-d815db304f6d\" (UID: \"0fae4c62-7dbc-4a61-9964-d815db304f6d\") "
Jul 31 02:38:41.655346 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.655202    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7-plugins-config-volume\") pod \"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7\" (UID: \"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7\") "
Jul 31 02:38:41.655511 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.655389    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b55133-3b5d-4124-921a-52eef9595441-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7b55133-3b5d-4124-921a-52eef9595441" (UID: "f7b55133-3b5d-4124-921a-52eef9595441"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:38:41.655511 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.655425    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fae4c62-7dbc-4a61-9964-d815db304f6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0fae4c62-7dbc-4a61-9964-d815db304f6d" (UID: "0fae4c62-7dbc-4a61-9964-d815db304f6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:38:41.655941 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.655790    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7" (UID: "c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:38:41.657387 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.657360    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b55133-3b5d-4124-921a-52eef9595441-kube-api-access-dg6lp" (OuterVolumeSpecName: "kube-api-access-dg6lp") pod "f7b55133-3b5d-4124-921a-52eef9595441" (UID: "f7b55133-3b5d-4124-921a-52eef9595441"). InnerVolumeSpecName "kube-api-access-dg6lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:38:41.657473 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.657403    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fae4c62-7dbc-4a61-9964-d815db304f6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0fae4c62-7dbc-4a61-9964-d815db304f6d" (UID: "0fae4c62-7dbc-4a61-9964-d815db304f6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:38:41.657473 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.657408    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7-kube-api-access-79dgw" (OuterVolumeSpecName: "kube-api-access-79dgw") pod "c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7" (UID: "c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7"). InnerVolumeSpecName "kube-api-access-79dgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:38:41.657473 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.657431    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fae4c62-7dbc-4a61-9964-d815db304f6d-kube-api-access-w7jft" (OuterVolumeSpecName: "kube-api-access-w7jft") pod "0fae4c62-7dbc-4a61-9964-d815db304f6d" (UID: "0fae4c62-7dbc-4a61-9964-d815db304f6d"). InnerVolumeSpecName "kube-api-access-w7jft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:38:41.756171 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.756122    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79dgw\" (UniqueName: \"kubernetes.io/projected/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7-kube-api-access-79dgw\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:38:41.756171 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.756142    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fae4c62-7dbc-4a61-9964-d815db304f6d-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:38:41.756171 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.756151    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg6lp\" (UniqueName: \"kubernetes.io/projected/f7b55133-3b5d-4124-921a-52eef9595441-kube-api-access-dg6lp\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:38:41.756171 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.756160    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7jft\" (UniqueName: \"kubernetes.io/projected/0fae4c62-7dbc-4a61-9964-d815db304f6d-kube-api-access-w7jft\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:38:41.756171 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.756168    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fae4c62-7dbc-4a61-9964-d815db304f6d-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:38:41.756363 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.756175    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:38:41.756363 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.756183    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b55133-3b5d-4124-921a-52eef9595441-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:38:41.763567 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:41.763513    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:38:42.437959 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.437910    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"
Jul 31 02:38:42.438301 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.437951    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f" event={"ID":"c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7","Type":"ContainerDied","Data":"bb712e34d3ee1fc2a905beaf3fd376856b6223c76c7800c69313cd1453df835e"}
Jul 31 02:38:42.438301 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.437999    2556 scope.go:117] "RemoveContainer" containerID="54e2ff68100f49f2d81372fda03a8d95ebeebc8a908aaf1172142597b1033845"
Jul 31 02:38:42.439438 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.439425    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"
Jul 31 02:38:42.439506 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.439446    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh" event={"ID":"f7b55133-3b5d-4124-921a-52eef9595441","Type":"ContainerDied","Data":"badb3919ac4227107ed9171c89900fff8e989c4a0cbf0159f469b4e308afdcc9"}
Jul 31 02:38:42.441006 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.440985    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn" event={"ID":"0fae4c62-7dbc-4a61-9964-d815db304f6d","Type":"ContainerDied","Data":"6d26d60a21be8dd3ae2f220607a622b4137ccb59056287ee1dc85c9d732ded45"}
Jul 31 02:38:42.441110 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.441011    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"
Jul 31 02:38:42.447275 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.447259    2556 scope.go:117] "RemoveContainer" containerID="ad4773253748507217a3194b896eb1a2451715947f0207a8f1e964a456da9e50"
Jul 31 02:38:42.454478 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.454461    2556 scope.go:117] "RemoveContainer" containerID="a3f1f2b62efa95729293e9ca151570df7aa6c93c169cc5ccee291fdb6445086f"
Jul 31 02:38:42.460638 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.460617    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"]
Jul 31 02:38:42.463101 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.463083    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e0efa5-6dc999c4b4-gqlrh"]
Jul 31 02:38:42.473263 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.473241    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"]
Jul 31 02:38:42.483262 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.483239    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0efa5-86994484cd-8qwrn"]
Jul 31 02:38:42.495165 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.495144    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"]
Jul 31 02:38:42.500109 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:42.500089    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e0efa5-64d7bff99d-xzm4f"]
Jul 31 02:38:44.150897 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:44.150867    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fae4c62-7dbc-4a61-9964-d815db304f6d" path="/var/lib/kubelet/pods/0fae4c62-7dbc-4a61-9964-d815db304f6d/volumes"
Jul 31 02:38:44.151240 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:44.151180    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7" path="/var/lib/kubelet/pods/c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7/volumes"
Jul 31 02:38:44.151458 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:44.151445    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7b55133-3b5d-4124-921a-52eef9595441" path="/var/lib/kubelet/pods/f7b55133-3b5d-4124-921a-52eef9595441/volumes"
Jul 31 02:38:46.781080 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:38:46.781046    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:22.985999 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:22.985971    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:23.848381 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:23.848355    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:29.437865 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:29.437832    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:34.952772 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:34.952728    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:35.052986 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:35.052945    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:35.269369 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:35.269295    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:35.553497 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:35.553430    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:46.084414 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.084372    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"]
Jul 31 02:39:46.084868 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.084617    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl" podUID="b910e789-7d08-4378-89b1-b9e8f9aef317" containerName="maas-api" containerID="cri-o://487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4" gracePeriod=30
Jul 31 02:39:46.121068 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.121046    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:46.152576 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.152536    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"]
Jul 31 02:39:46.152791 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.152747    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w" podUID="0dffbfa0-415d-4c07-bd36-ca787db74c57" containerName="payload-processing" containerID="cri-o://e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790" gracePeriod=30
Jul 31 02:39:46.158341 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.158296    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"]
Jul 31 02:39:46.158517 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.158468    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv" podUID="5d388446-1b53-4e27-ba67-6afa0eefbd9e" containerName="payload-pre-processing" containerID="cri-o://c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798" gracePeriod=30
Jul 31 02:39:46.223119 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.223079    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:46.303842 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.303742    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:46.366428 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.366301    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:46.414580 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.414528    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:39:46.441246 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.441223    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"
Jul 31 02:39:46.474922 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.474899    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"
Jul 31 02:39:46.508023 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.507997    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t2rw\" (UniqueName: \"kubernetes.io/projected/0dffbfa0-415d-4c07-bd36-ca787db74c57-kube-api-access-2t2rw\") pod \"0dffbfa0-415d-4c07-bd36-ca787db74c57\" (UID: \"0dffbfa0-415d-4c07-bd36-ca787db74c57\") "
Jul 31 02:39:46.508145 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.508044    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dffbfa0-415d-4c07-bd36-ca787db74c57-plugins-config-volume\") pod \"0dffbfa0-415d-4c07-bd36-ca787db74c57\" (UID: \"0dffbfa0-415d-4c07-bd36-ca787db74c57\") "
Jul 31 02:39:46.508145 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.508074    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9v28\" (UniqueName: \"kubernetes.io/projected/b910e789-7d08-4378-89b1-b9e8f9aef317-kube-api-access-s9v28\") pod \"b910e789-7d08-4378-89b1-b9e8f9aef317\" (UID: \"b910e789-7d08-4378-89b1-b9e8f9aef317\") "
Jul 31 02:39:46.508145 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.508116    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b910e789-7d08-4378-89b1-b9e8f9aef317-maas-api-tls\") pod \"b910e789-7d08-4378-89b1-b9e8f9aef317\" (UID: \"b910e789-7d08-4378-89b1-b9e8f9aef317\") "
Jul 31 02:39:46.508145 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.508139    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b910e789-7d08-4378-89b1-b9e8f9aef317-openshift-service-ca\") pod \"b910e789-7d08-4378-89b1-b9e8f9aef317\" (UID: \"b910e789-7d08-4378-89b1-b9e8f9aef317\") "
Jul 31 02:39:46.508468 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.508440    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dffbfa0-415d-4c07-bd36-ca787db74c57-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0dffbfa0-415d-4c07-bd36-ca787db74c57" (UID: "0dffbfa0-415d-4c07-bd36-ca787db74c57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:39:46.508643 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.508617    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b910e789-7d08-4378-89b1-b9e8f9aef317-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b910e789-7d08-4378-89b1-b9e8f9aef317" (UID: "b910e789-7d08-4378-89b1-b9e8f9aef317"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:39:46.510360 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.510340    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b910e789-7d08-4378-89b1-b9e8f9aef317-kube-api-access-s9v28" (OuterVolumeSpecName: "kube-api-access-s9v28") pod "b910e789-7d08-4378-89b1-b9e8f9aef317" (UID: "b910e789-7d08-4378-89b1-b9e8f9aef317"). InnerVolumeSpecName "kube-api-access-s9v28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:39:46.510634 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.510609    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b910e789-7d08-4378-89b1-b9e8f9aef317-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b910e789-7d08-4378-89b1-b9e8f9aef317" (UID: "b910e789-7d08-4378-89b1-b9e8f9aef317"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:39:46.510714 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.510640    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dffbfa0-415d-4c07-bd36-ca787db74c57-kube-api-access-2t2rw" (OuterVolumeSpecName: "kube-api-access-2t2rw") pod "0dffbfa0-415d-4c07-bd36-ca787db74c57" (UID: "0dffbfa0-415d-4c07-bd36-ca787db74c57"). InnerVolumeSpecName "kube-api-access-2t2rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:39:46.609347 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.609326    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m2kz\" (UniqueName: \"kubernetes.io/projected/5d388446-1b53-4e27-ba67-6afa0eefbd9e-kube-api-access-6m2kz\") pod \"5d388446-1b53-4e27-ba67-6afa0eefbd9e\" (UID: \"5d388446-1b53-4e27-ba67-6afa0eefbd9e\") "
Jul 31 02:39:46.609447 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.609436    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d388446-1b53-4e27-ba67-6afa0eefbd9e-plugins-config-volume\") pod \"5d388446-1b53-4e27-ba67-6afa0eefbd9e\" (UID: \"5d388446-1b53-4e27-ba67-6afa0eefbd9e\") "
Jul 31 02:39:46.609608 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.609596    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t2rw\" (UniqueName: \"kubernetes.io/projected/0dffbfa0-415d-4c07-bd36-ca787db74c57-kube-api-access-2t2rw\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:46.609661 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.609611    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dffbfa0-415d-4c07-bd36-ca787db74c57-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:46.609661 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.609620    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9v28\" (UniqueName: \"kubernetes.io/projected/b910e789-7d08-4378-89b1-b9e8f9aef317-kube-api-access-s9v28\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:46.609661 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.609628    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b910e789-7d08-4378-89b1-b9e8f9aef317-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:46.609661 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.609636    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b910e789-7d08-4378-89b1-b9e8f9aef317-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:46.609835 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.609772    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d388446-1b53-4e27-ba67-6afa0eefbd9e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d388446-1b53-4e27-ba67-6afa0eefbd9e" (UID: "5d388446-1b53-4e27-ba67-6afa0eefbd9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:39:46.611115 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.611099    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d388446-1b53-4e27-ba67-6afa0eefbd9e-kube-api-access-6m2kz" (OuterVolumeSpecName: "kube-api-access-6m2kz") pod "5d388446-1b53-4e27-ba67-6afa0eefbd9e" (UID: "5d388446-1b53-4e27-ba67-6afa0eefbd9e"). InnerVolumeSpecName "kube-api-access-6m2kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:39:46.710479 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.710432    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d388446-1b53-4e27-ba67-6afa0eefbd9e-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:46.710479 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.710452    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m2kz\" (UniqueName: \"kubernetes.io/projected/5d388446-1b53-4e27-ba67-6afa0eefbd9e-kube-api-access-6m2kz\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:46.726567 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.726528    2556 generic.go:356] "Generic (PLEG): container finished" podID="5d388446-1b53-4e27-ba67-6afa0eefbd9e" containerID="c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798" exitCode=0
Jul 31 02:39:46.726655 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.726600    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"
Jul 31 02:39:46.726655 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.726625    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv" event={"ID":"5d388446-1b53-4e27-ba67-6afa0eefbd9e","Type":"ContainerDied","Data":"c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798"}
Jul 31 02:39:46.726761 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.726683    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv" event={"ID":"5d388446-1b53-4e27-ba67-6afa0eefbd9e","Type":"ContainerDied","Data":"9909ae7f745d909128d8fc0246f3dc5d583fad688bd42bc3b54d35553e9b3ec4"}
Jul 31 02:39:46.726761 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.726716    2556 scope.go:117] "RemoveContainer" containerID="c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798"
Jul 31 02:39:46.727928 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.727905    2556 generic.go:356] "Generic (PLEG): container finished" podID="0dffbfa0-415d-4c07-bd36-ca787db74c57" containerID="e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790" exitCode=0
Jul 31 02:39:46.728004 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.727942    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"
Jul 31 02:39:46.728060 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.727997    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w" event={"ID":"0dffbfa0-415d-4c07-bd36-ca787db74c57","Type":"ContainerDied","Data":"e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790"}
Jul 31 02:39:46.728060 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.728042    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w" event={"ID":"0dffbfa0-415d-4c07-bd36-ca787db74c57","Type":"ContainerDied","Data":"6bce193fa3eaaa6308abe0e7a6a8b3a3c2472d14db6620ce924094c21c1c4b13"}
Jul 31 02:39:46.729665 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.729642    2556 generic.go:356] "Generic (PLEG): container finished" podID="b910e789-7d08-4378-89b1-b9e8f9aef317" containerID="487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4" exitCode=0
Jul 31 02:39:46.729762 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.729688    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl" event={"ID":"b910e789-7d08-4378-89b1-b9e8f9aef317","Type":"ContainerDied","Data":"487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4"}
Jul 31 02:39:46.729762 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.729696    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"
Jul 31 02:39:46.729762 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.729722    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl" event={"ID":"b910e789-7d08-4378-89b1-b9e8f9aef317","Type":"ContainerDied","Data":"22ef18f1867529b95b39f4e594eaf4852f2e57776d1cd6eae5ed4e7def322a48"}
Jul 31 02:39:46.730701 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.730671    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:46.736801 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.736776    2556 scope.go:117] "RemoveContainer" containerID="c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798"
Jul 31 02:39:46.737077 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:39:46.737052    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798\": container with ID starting with c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798 not found: ID does not exist" containerID="c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798"
Jul 31 02:39:46.737138 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.737089    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798"} err="failed to get container status \"c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798\": rpc error: code = NotFound desc = could not find container \"c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798\": container with ID starting with c199b68a76f8a19636ca5c706bd93dcf98f380ef4ed9d01a535361725d7fe798 not found: ID does not exist"
Jul 31 02:39:46.737138 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.737114    2556 scope.go:117] "RemoveContainer" containerID="e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790"
Jul 31 02:39:46.745186 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.745170    2556 scope.go:117] "RemoveContainer" containerID="e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790"
Jul 31 02:39:46.745408 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:39:46.745394    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790\": container with ID starting with e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790 not found: ID does not exist" containerID="e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790"
Jul 31 02:39:46.745454 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.745413    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790"} err="failed to get container status \"e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790\": rpc error: code = NotFound desc = could not find container \"e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790\": container with ID starting with e30dd5f2a83aced757b94237c1b09e5e69b623808fb88fa36d79b460f13c1790 not found: ID does not exist"
Jul 31 02:39:46.745454 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.745426    2556 scope.go:117] "RemoveContainer" containerID="487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4"
Jul 31 02:39:46.751040 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.751022    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"]
Jul 31 02:39:46.753085 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.753063    2556 scope.go:117] "RemoveContainer" containerID="487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4"
Jul 31 02:39:46.753515 ip-10-0-141-142 kubenswrapper[2556]: E0731 02:39:46.753487    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4\": container with ID starting with 487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4 not found: ID does not exist" containerID="487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4"
Jul 31 02:39:46.753612 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.753523    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4"} err="failed to get container status \"487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4\": rpc error: code = NotFound desc = could not find container \"487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4\": container with ID starting with 487a132dd2912fa54759b2a022758f57cd0ce0f58f4d26c08afc10fa47181cc4 not found: ID does not exist"
Jul 31 02:39:46.755100 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.755083    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3dc29c-55dc879c9b-p94lv"]
Jul 31 02:39:46.764494 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.764475    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"]
Jul 31 02:39:46.768327 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.768311    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3dc29c-6dd44bbb6c-nd7cl"]
Jul 31 02:39:46.778624 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.778603    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"]
Jul 31 02:39:46.782128 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:46.782111    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3dc29c-6f66d7f95b-z8s9w"]
Jul 31 02:39:48.151529 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:48.151495    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dffbfa0-415d-4c07-bd36-ca787db74c57" path="/var/lib/kubelet/pods/0dffbfa0-415d-4c07-bd36-ca787db74c57/volumes"
Jul 31 02:39:48.152012 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:48.151964    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d388446-1b53-4e27-ba67-6afa0eefbd9e" path="/var/lib/kubelet/pods/5d388446-1b53-4e27-ba67-6afa0eefbd9e/volumes"
Jul 31 02:39:48.152345 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:48.152325    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b910e789-7d08-4378-89b1-b9e8f9aef317" path="/var/lib/kubelet/pods/b910e789-7d08-4378-89b1-b9e8f9aef317/volumes"
Jul 31 02:39:52.561076 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:52.561051    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:52.629512 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:52.629485    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:52.875341 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:52.875270    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:53.130510 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:53.130439    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:57.493959 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.493921    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"]
Jul 31 02:39:57.494433 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.494148    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs" podUID="e4ce3593-7284-47f1-8546-f7f401a7a9f0" containerName="maas-api" containerID="cri-o://96b8a25d44a09bcf751024c58d1440c4f7aed7f41bf452831887c8e8acbeee5e" gracePeriod=30
Jul 31 02:39:57.522459 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.522437    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp"]
Jul 31 02:39:57.525720 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.525698    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:57.527373 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.527351    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-2aed0b-29757750-46qzp"]
Jul 31 02:39:57.554326 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.554305    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"]
Jul 31 02:39:57.554509 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.554487    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps" podUID="9b5d2a85-4ea5-423d-9a84-ec340fbe73ff" containerName="payload-processing" containerID="cri-o://e9e29ec3a2edddc86df14bb63b262d22273a9eaadaac79fdaeda5b42c6a8ba5a" gracePeriod=30
Jul 31 02:39:57.560200 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.560178    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"]
Jul 31 02:39:57.560361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.560336    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m" podUID="8e2b2759-8e22-4886-ab27-429992953694" containerName="payload-pre-processing" containerID="cri-o://8934b3091233e8a7035d165cb742d990b32ca9d761b20df6878298d7edee5503" gracePeriod=30
Jul 31 02:39:57.599062 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.599033    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:57.660785 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.660758    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:57.718411 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.718348    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:57.785865 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.785838    2556 generic.go:356] "Generic (PLEG): container finished" podID="e4ce3593-7284-47f1-8546-f7f401a7a9f0" containerID="96b8a25d44a09bcf751024c58d1440c4f7aed7f41bf452831887c8e8acbeee5e" exitCode=0
Jul 31 02:39:57.786020 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.785909    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs" event={"ID":"e4ce3593-7284-47f1-8546-f7f401a7a9f0","Type":"ContainerDied","Data":"96b8a25d44a09bcf751024c58d1440c4f7aed7f41bf452831887c8e8acbeee5e"}
Jul 31 02:39:57.787859 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.787819    2556 generic.go:356] "Generic (PLEG): container finished" podID="9b5d2a85-4ea5-423d-9a84-ec340fbe73ff" containerID="e9e29ec3a2edddc86df14bb63b262d22273a9eaadaac79fdaeda5b42c6a8ba5a" exitCode=0
Jul 31 02:39:57.787990 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.787868    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps" event={"ID":"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff","Type":"ContainerDied","Data":"e9e29ec3a2edddc86df14bb63b262d22273a9eaadaac79fdaeda5b42c6a8ba5a"}
Jul 31 02:39:57.789714 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.789689    2556 generic.go:356] "Generic (PLEG): container finished" podID="8e2b2759-8e22-4886-ab27-429992953694" containerID="8934b3091233e8a7035d165cb742d990b32ca9d761b20df6878298d7edee5503" exitCode=0
Jul 31 02:39:57.789814 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.789793    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m" event={"ID":"8e2b2759-8e22-4886-ab27-429992953694","Type":"ContainerDied","Data":"8934b3091233e8a7035d165cb742d990b32ca9d761b20df6878298d7edee5503"}
Jul 31 02:39:57.843295 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.843272    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"
Jul 31 02:39:57.846019 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.846001    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"
Jul 31 02:39:57.848920 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.848900    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:39:57.994866 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.994828    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csgr5\" (UniqueName: \"kubernetes.io/projected/e4ce3593-7284-47f1-8546-f7f401a7a9f0-kube-api-access-csgr5\") pod \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\" (UID: \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\") "
Jul 31 02:39:57.995028 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.994899    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ce3593-7284-47f1-8546-f7f401a7a9f0-openshift-service-ca\") pod \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\" (UID: \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\") "
Jul 31 02:39:57.995028 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.994992    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knff9\" (UniqueName: \"kubernetes.io/projected/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff-kube-api-access-knff9\") pod \"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff\" (UID: \"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff\") "
Jul 31 02:39:57.995157 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.995041    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnnpp\" (UniqueName: \"kubernetes.io/projected/8e2b2759-8e22-4886-ab27-429992953694-kube-api-access-jnnpp\") pod \"8e2b2759-8e22-4886-ab27-429992953694\" (UID: \"8e2b2759-8e22-4886-ab27-429992953694\") "
Jul 31 02:39:57.995157 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.995060    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b2759-8e22-4886-ab27-429992953694-plugins-config-volume\") pod \"8e2b2759-8e22-4886-ab27-429992953694\" (UID: \"8e2b2759-8e22-4886-ab27-429992953694\") "
Jul 31 02:39:57.995157 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.995088    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ce3593-7284-47f1-8546-f7f401a7a9f0-maas-api-tls\") pod \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\" (UID: \"e4ce3593-7284-47f1-8546-f7f401a7a9f0\") "
Jul 31 02:39:57.995157 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.995103    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff-plugins-config-volume\") pod \"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff\" (UID: \"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff\") "
Jul 31 02:39:57.995343 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.995212    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ce3593-7284-47f1-8546-f7f401a7a9f0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4ce3593-7284-47f1-8546-f7f401a7a9f0" (UID: "e4ce3593-7284-47f1-8546-f7f401a7a9f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:39:57.995501 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.995473    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ce3593-7284-47f1-8546-f7f401a7a9f0-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:57.995638 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.995472    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2b2759-8e22-4886-ab27-429992953694-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e2b2759-8e22-4886-ab27-429992953694" (UID: "8e2b2759-8e22-4886-ab27-429992953694"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:39:57.995638 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.995511    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b5d2a85-4ea5-423d-9a84-ec340fbe73ff" (UID: "9b5d2a85-4ea5-423d-9a84-ec340fbe73ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:39:57.997145 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.997118    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ce3593-7284-47f1-8546-f7f401a7a9f0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4ce3593-7284-47f1-8546-f7f401a7a9f0" (UID: "e4ce3593-7284-47f1-8546-f7f401a7a9f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:39:57.997145 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.997130    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2b2759-8e22-4886-ab27-429992953694-kube-api-access-jnnpp" (OuterVolumeSpecName: "kube-api-access-jnnpp") pod "8e2b2759-8e22-4886-ab27-429992953694" (UID: "8e2b2759-8e22-4886-ab27-429992953694"). InnerVolumeSpecName "kube-api-access-jnnpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:39:57.997491 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.997473    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff-kube-api-access-knff9" (OuterVolumeSpecName: "kube-api-access-knff9") pod "9b5d2a85-4ea5-423d-9a84-ec340fbe73ff" (UID: "9b5d2a85-4ea5-423d-9a84-ec340fbe73ff"). InnerVolumeSpecName "kube-api-access-knff9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:39:57.997547 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:57.997502    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ce3593-7284-47f1-8546-f7f401a7a9f0-kube-api-access-csgr5" (OuterVolumeSpecName: "kube-api-access-csgr5") pod "e4ce3593-7284-47f1-8546-f7f401a7a9f0" (UID: "e4ce3593-7284-47f1-8546-f7f401a7a9f0"). InnerVolumeSpecName "kube-api-access-csgr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:39:58.096371 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.096340    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knff9\" (UniqueName: \"kubernetes.io/projected/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff-kube-api-access-knff9\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:58.096371 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.096371    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnnpp\" (UniqueName: \"kubernetes.io/projected/8e2b2759-8e22-4886-ab27-429992953694-kube-api-access-jnnpp\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:58.096506 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.096385    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b2759-8e22-4886-ab27-429992953694-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:58.096506 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.096396    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ce3593-7284-47f1-8546-f7f401a7a9f0-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:58.096506 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.096408    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:58.096506 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.096420    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csgr5\" (UniqueName: \"kubernetes.io/projected/e4ce3593-7284-47f1-8546-f7f401a7a9f0-kube-api-access-csgr5\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:39:58.140092 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.140065    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:39:58.150577 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.150538    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fddcd1d-9836-4aee-8f30-d0b3d2220b60" path="/var/lib/kubelet/pods/7fddcd1d-9836-4aee-8f30-d0b3d2220b60/volumes"
Jul 31 02:39:58.795287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.795230    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"
Jul 31 02:39:58.795287 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.795224    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs" event={"ID":"e4ce3593-7284-47f1-8546-f7f401a7a9f0","Type":"ContainerDied","Data":"a5e8166c11ff6ac84a11fe90c3260f6d1cb44375864a786793c2f1d36eac3cb2"}
Jul 31 02:39:58.795717 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.795342    2556 scope.go:117] "RemoveContainer" containerID="96b8a25d44a09bcf751024c58d1440c4f7aed7f41bf452831887c8e8acbeee5e"
Jul 31 02:39:58.796657 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.796551    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"
Jul 31 02:39:58.796657 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.796580    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps" event={"ID":"9b5d2a85-4ea5-423d-9a84-ec340fbe73ff","Type":"ContainerDied","Data":"0e27be60ee47830e972a47418f82b976e68ebdc996edaa75c7a229ba6ef9a4f7"}
Jul 31 02:39:58.797935 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.797920    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"
Jul 31 02:39:58.798031 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.797947    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m" event={"ID":"8e2b2759-8e22-4886-ab27-429992953694","Type":"ContainerDied","Data":"ea2df13c27294c8c82fb06ef9101a7d78035704c688caa158d4b79471e2cd46d"}
Jul 31 02:39:58.804381 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.804367    2556 scope.go:117] "RemoveContainer" containerID="e9e29ec3a2edddc86df14bb63b262d22273a9eaadaac79fdaeda5b42c6a8ba5a"
Jul 31 02:39:58.811265 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.811250    2556 scope.go:117] "RemoveContainer" containerID="8934b3091233e8a7035d165cb742d990b32ca9d761b20df6878298d7edee5503"
Jul 31 02:39:58.814250 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.814234    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"]
Jul 31 02:39:58.817304 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.817285    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2aed0b-59d97db44b-992ps"]
Jul 31 02:39:58.826407 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.826370    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"]
Jul 31 02:39:58.827963 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.827942    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2aed0b-664f95567b-tgn5m"]
Jul 31 02:39:58.837505 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.837484    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"]
Jul 31 02:39:58.842671 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:39:58.842652    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2aed0b-8665b94d98-8vrxs"]
Jul 31 02:40:00.150054 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:00.150020    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e2b2759-8e22-4886-ab27-429992953694" path="/var/lib/kubelet/pods/8e2b2759-8e22-4886-ab27-429992953694/volumes"
Jul 31 02:40:00.150485 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:00.150451    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b5d2a85-4ea5-423d-9a84-ec340fbe73ff" path="/var/lib/kubelet/pods/9b5d2a85-4ea5-423d-9a84-ec340fbe73ff/volumes"
Jul 31 02:40:00.151007 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:00.150973    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4ce3593-7284-47f1-8546-f7f401a7a9f0" path="/var/lib/kubelet/pods/e4ce3593-7284-47f1-8546-f7f401a7a9f0/volumes"
Jul 31 02:40:01.573772 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:01.573731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:04.470798 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:04.470771    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:04.575547 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:04.575519    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:04.807908 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:04.807882    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:05.068367 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.068282    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:05.201092 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201068    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl"]
Jul 31 02:40:05.201437 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201420    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7" containerName="payload-processing"
Jul 31 02:40:05.201519 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201441    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7" containerName="payload-processing"
Jul 31 02:40:05.201519 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201453    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b910e789-7d08-4378-89b1-b9e8f9aef317" containerName="maas-api"
Jul 31 02:40:05.201519 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201461    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b910e789-7d08-4378-89b1-b9e8f9aef317" containerName="maas-api"
Jul 31 02:40:05.201519 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201474    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4ce3593-7284-47f1-8546-f7f401a7a9f0" containerName="maas-api"
Jul 31 02:40:05.201519 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201483    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4ce3593-7284-47f1-8546-f7f401a7a9f0" containerName="maas-api"
Jul 31 02:40:05.201519 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201492    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fae4c62-7dbc-4a61-9964-d815db304f6d" containerName="maas-api"
Jul 31 02:40:05.201519 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201500    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fae4c62-7dbc-4a61-9964-d815db304f6d" containerName="maas-api"
Jul 31 02:40:05.201519 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201510    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39ecabc1-4e4b-4d53-b805-6c46c5890606" containerName="payload-processing"
Jul 31 02:40:05.201519 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201518    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="39ecabc1-4e4b-4d53-b805-6c46c5890606" containerName="payload-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201530    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b55133-3b5d-4124-921a-52eef9595441" containerName="payload-pre-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201538    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b55133-3b5d-4124-921a-52eef9595441" containerName="payload-pre-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201552    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dffbfa0-415d-4c07-bd36-ca787db74c57" containerName="payload-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201581    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dffbfa0-415d-4c07-bd36-ca787db74c57" containerName="payload-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201603    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e2b2759-8e22-4886-ab27-429992953694" containerName="payload-pre-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201611    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e2b2759-8e22-4886-ab27-429992953694" containerName="payload-pre-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201622    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8" containerName="maas-api"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201629    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8" containerName="maas-api"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201640    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d388446-1b53-4e27-ba67-6afa0eefbd9e" containerName="payload-pre-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201649    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d388446-1b53-4e27-ba67-6afa0eefbd9e" containerName="payload-pre-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201658    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b5d2a85-4ea5-423d-9a84-ec340fbe73ff" containerName="payload-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201666    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b5d2a85-4ea5-423d-9a84-ec340fbe73ff" containerName="payload-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201740    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e2b2759-8e22-4886-ab27-429992953694" containerName="payload-pre-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201751    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b55133-3b5d-4124-921a-52eef9595441" containerName="payload-pre-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201759    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fae4c62-7dbc-4a61-9964-d815db304f6d" containerName="maas-api"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201766    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b910e789-7d08-4378-89b1-b9e8f9aef317" containerName="maas-api"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201774    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b5d2a85-4ea5-423d-9a84-ec340fbe73ff" containerName="payload-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201781    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ff9f8f-526d-4445-b3d4-9ccf64c82cb7" containerName="payload-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201789    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4ce3593-7284-47f1-8546-f7f401a7a9f0" containerName="maas-api"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201796    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d388446-1b53-4e27-ba67-6afa0eefbd9e" containerName="payload-pre-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201807    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dffbfa0-415d-4c07-bd36-ca787db74c57" containerName="payload-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201814    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="39ecabc1-4e4b-4d53-b805-6c46c5890606" containerName="payload-processing"
Jul 31 02:40:05.201953 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.201822    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd8cff8-2dd4-4aaa-bc52-e6c5491547b8" containerName="maas-api"
Jul 31 02:40:05.206499 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.206479    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl"
Jul 31 02:40:05.209008 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.208992    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lh69\""
Jul 31 02:40:05.212631 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.212605    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl"]
Jul 31 02:40:05.347449 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.347387    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293bbde2-3cda-4954-84f1-6a3a11e028a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl\" (UID: \"293bbde2-3cda-4954-84f1-6a3a11e028a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl"
Jul 31 02:40:05.448250 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.448226    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293bbde2-3cda-4954-84f1-6a3a11e028a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl\" (UID: \"293bbde2-3cda-4954-84f1-6a3a11e028a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl"
Jul 31 02:40:05.448763 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.448747    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293bbde2-3cda-4954-84f1-6a3a11e028a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl\" (UID: \"293bbde2-3cda-4954-84f1-6a3a11e028a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl"
Jul 31 02:40:05.517033 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.517011    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl"
Jul 31 02:40:05.647157 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.647134    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl"]
Jul 31 02:40:05.649532 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:40:05.649505    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293bbde2_3cda_4954_84f1_6a3a11e028a6.slice/crio-f24b265fad8dba6ed86f9042c146666e66e2a0f5c261703aa89c56286ab80b7e WatchSource:0}: Error finding container f24b265fad8dba6ed86f9042c146666e66e2a0f5c261703aa89c56286ab80b7e: Status 404 returned error can't find the container with id f24b265fad8dba6ed86f9042c146666e66e2a0f5c261703aa89c56286ab80b7e
Jul 31 02:40:05.829429 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:05.829407    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl" event={"ID":"293bbde2-3cda-4954-84f1-6a3a11e028a6","Type":"ContainerStarted","Data":"f24b265fad8dba6ed86f9042c146666e66e2a0f5c261703aa89c56286ab80b7e"}
Jul 31 02:40:06.835802 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:06.835767    2556 generic.go:356] "Generic (PLEG): container finished" podID="293bbde2-3cda-4954-84f1-6a3a11e028a6" containerID="b50b2805b1f2d407a95e3c9d8e61c0f8855a826a926e26e1c44cbb722f4a123e" exitCode=0
Jul 31 02:40:06.836162 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:06.835853    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl" event={"ID":"293bbde2-3cda-4954-84f1-6a3a11e028a6","Type":"ContainerDied","Data":"b50b2805b1f2d407a95e3c9d8e61c0f8855a826a926e26e1c44cbb722f4a123e"}
Jul 31 02:40:07.966600 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:07.966578    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl"
Jul 31 02:40:08.068740 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:08.068713    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293bbde2-3cda-4954-84f1-6a3a11e028a6-maas-api-service-ca\") pod \"293bbde2-3cda-4954-84f1-6a3a11e028a6\" (UID: \"293bbde2-3cda-4954-84f1-6a3a11e028a6\") "
Jul 31 02:40:08.069021 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:08.069002    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293bbde2-3cda-4954-84f1-6a3a11e028a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "293bbde2-3cda-4954-84f1-6a3a11e028a6" (UID: "293bbde2-3cda-4954-84f1-6a3a11e028a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:40:08.169204 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:08.169185    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293bbde2-3cda-4954-84f1-6a3a11e028a6-maas-api-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:40:08.846861 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:08.846838    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl"
Jul 31 02:40:08.847005 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:08.846859    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl" event={"ID":"293bbde2-3cda-4954-84f1-6a3a11e028a6","Type":"ContainerDied","Data":"f24b265fad8dba6ed86f9042c146666e66e2a0f5c261703aa89c56286ab80b7e"}
Jul 31 02:40:08.847005 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:08.846931    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f24b265fad8dba6ed86f9042c146666e66e2a0f5c261703aa89c56286ab80b7e"
Jul 31 02:40:15.649229 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.649151    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"]
Jul 31 02:40:15.649672 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.649398    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc" podUID="054b0c8e-37b7-4837-8b3d-5a6a422ed171" containerName="maas-api" containerID="cri-o://4e301479f08431bec62583bd0cf6b834e55d5ace4e08fac1c8f82329a93d4256" gracePeriod=30
Jul 31 02:40:15.694045 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.694021    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-01a44c-29757750-jhs48"]
Jul 31 02:40:15.697424 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.697405    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-01a44c-29757750-jhs48"]
Jul 31 02:40:15.701428 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.701409    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:15.732816 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.732785    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"]
Jul 31 02:40:15.733005 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.732983    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9" podUID="92de2058-3d2c-4b20-b78f-cff131643350" containerName="payload-processing" containerID="cri-o://8a81cbcec778f8b8c1eb65db5b8fef316dfc0adb3cf4d87ee9fc558c26d873a5" gracePeriod=30
Jul 31 02:40:15.740743 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.740720    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"]
Jul 31 02:40:15.740903 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.740882    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n" podUID="cf4511df-478e-4cda-9b83-e6fac0490d3f" containerName="payload-pre-processing" containerID="cri-o://88602f035ca93b90967a734a4b3f8a1fa296967fd6d174d34e58e9b5c3fba63e" gracePeriod=30
Jul 31 02:40:15.807106 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.807076    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:15.880475 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.880446    2556 generic.go:356] "Generic (PLEG): container finished" podID="92de2058-3d2c-4b20-b78f-cff131643350" containerID="8a81cbcec778f8b8c1eb65db5b8fef316dfc0adb3cf4d87ee9fc558c26d873a5" exitCode=0
Jul 31 02:40:15.880669 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.880520    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9" event={"ID":"92de2058-3d2c-4b20-b78f-cff131643350","Type":"ContainerDied","Data":"8a81cbcec778f8b8c1eb65db5b8fef316dfc0adb3cf4d87ee9fc558c26d873a5"}
Jul 31 02:40:15.881988 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.881967    2556 generic.go:356] "Generic (PLEG): container finished" podID="054b0c8e-37b7-4837-8b3d-5a6a422ed171" containerID="4e301479f08431bec62583bd0cf6b834e55d5ace4e08fac1c8f82329a93d4256" exitCode=0
Jul 31 02:40:15.882107 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.882093    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc" event={"ID":"054b0c8e-37b7-4837-8b3d-5a6a422ed171","Type":"ContainerDied","Data":"4e301479f08431bec62583bd0cf6b834e55d5ace4e08fac1c8f82329a93d4256"}
Jul 31 02:40:15.883606 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.883579    2556 generic.go:356] "Generic (PLEG): container finished" podID="cf4511df-478e-4cda-9b83-e6fac0490d3f" containerID="88602f035ca93b90967a734a4b3f8a1fa296967fd6d174d34e58e9b5c3fba63e" exitCode=0
Jul 31 02:40:15.883709 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.883633    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n" event={"ID":"cf4511df-478e-4cda-9b83-e6fac0490d3f","Type":"ContainerDied","Data":"88602f035ca93b90967a734a4b3f8a1fa296967fd6d174d34e58e9b5c3fba63e"}
Jul 31 02:40:15.897845 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.897808    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:15.939034 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.939005    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl"]
Jul 31 02:40:15.947782 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.947684    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-01a44c-jpbhl"]
Jul 31 02:40:15.991613 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:15.991579    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"
Jul 31 02:40:16.025010 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.024927    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:16.029546 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.029527    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:40:16.037998 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.037981    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"
Jul 31 02:40:16.127133 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.127113    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054b0c8e-37b7-4837-8b3d-5a6a422ed171-openshift-service-ca\") pod \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\" (UID: \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\") "
Jul 31 02:40:16.127133 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.127140    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054b0c8e-37b7-4837-8b3d-5a6a422ed171-maas-api-tls\") pod \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\" (UID: \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\") "
Jul 31 02:40:16.127326 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.127177    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4511df-478e-4cda-9b83-e6fac0490d3f-plugins-config-volume\") pod \"cf4511df-478e-4cda-9b83-e6fac0490d3f\" (UID: \"cf4511df-478e-4cda-9b83-e6fac0490d3f\") "
Jul 31 02:40:16.127326 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.127231    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2drzd\" (UniqueName: \"kubernetes.io/projected/92de2058-3d2c-4b20-b78f-cff131643350-kube-api-access-2drzd\") pod \"92de2058-3d2c-4b20-b78f-cff131643350\" (UID: \"92de2058-3d2c-4b20-b78f-cff131643350\") "
Jul 31 02:40:16.127326 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.127268    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7crb2\" (UniqueName: \"kubernetes.io/projected/cf4511df-478e-4cda-9b83-e6fac0490d3f-kube-api-access-7crb2\") pod \"cf4511df-478e-4cda-9b83-e6fac0490d3f\" (UID: \"cf4511df-478e-4cda-9b83-e6fac0490d3f\") "
Jul 31 02:40:16.127326 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.127304    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92de2058-3d2c-4b20-b78f-cff131643350-plugins-config-volume\") pod \"92de2058-3d2c-4b20-b78f-cff131643350\" (UID: \"92de2058-3d2c-4b20-b78f-cff131643350\") "
Jul 31 02:40:16.127527 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.127338    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wzxf\" (UniqueName: \"kubernetes.io/projected/054b0c8e-37b7-4837-8b3d-5a6a422ed171-kube-api-access-9wzxf\") pod \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\" (UID: \"054b0c8e-37b7-4837-8b3d-5a6a422ed171\") "
Jul 31 02:40:16.127527 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.127493    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054b0c8e-37b7-4837-8b3d-5a6a422ed171-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "054b0c8e-37b7-4837-8b3d-5a6a422ed171" (UID: "054b0c8e-37b7-4837-8b3d-5a6a422ed171"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:40:16.127648 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.127616    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054b0c8e-37b7-4837-8b3d-5a6a422ed171-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:40:16.127714 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.127686    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4511df-478e-4cda-9b83-e6fac0490d3f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf4511df-478e-4cda-9b83-e6fac0490d3f" (UID: "cf4511df-478e-4cda-9b83-e6fac0490d3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:40:16.127785 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.127749    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92de2058-3d2c-4b20-b78f-cff131643350-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "92de2058-3d2c-4b20-b78f-cff131643350" (UID: "92de2058-3d2c-4b20-b78f-cff131643350"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 02:40:16.129464 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.129435    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4511df-478e-4cda-9b83-e6fac0490d3f-kube-api-access-7crb2" (OuterVolumeSpecName: "kube-api-access-7crb2") pod "cf4511df-478e-4cda-9b83-e6fac0490d3f" (UID: "cf4511df-478e-4cda-9b83-e6fac0490d3f"). InnerVolumeSpecName "kube-api-access-7crb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:40:16.129464 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.129444    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054b0c8e-37b7-4837-8b3d-5a6a422ed171-kube-api-access-9wzxf" (OuterVolumeSpecName: "kube-api-access-9wzxf") pod "054b0c8e-37b7-4837-8b3d-5a6a422ed171" (UID: "054b0c8e-37b7-4837-8b3d-5a6a422ed171"). InnerVolumeSpecName "kube-api-access-9wzxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:40:16.129604 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.129456    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054b0c8e-37b7-4837-8b3d-5a6a422ed171-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "054b0c8e-37b7-4837-8b3d-5a6a422ed171" (UID: "054b0c8e-37b7-4837-8b3d-5a6a422ed171"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 02:40:16.129604 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.129462    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92de2058-3d2c-4b20-b78f-cff131643350-kube-api-access-2drzd" (OuterVolumeSpecName: "kube-api-access-2drzd") pod "92de2058-3d2c-4b20-b78f-cff131643350" (UID: "92de2058-3d2c-4b20-b78f-cff131643350"). InnerVolumeSpecName "kube-api-access-2drzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 02:40:16.150728 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.150703    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="293bbde2-3cda-4954-84f1-6a3a11e028a6" path="/var/lib/kubelet/pods/293bbde2-3cda-4954-84f1-6a3a11e028a6/volumes"
Jul 31 02:40:16.150948 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.150936    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac87b666-bb42-4ca9-abf1-645bc1fd2b7c" path="/var/lib/kubelet/pods/ac87b666-bb42-4ca9-abf1-645bc1fd2b7c/volumes"
Jul 31 02:40:16.228352 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.228328    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wzxf\" (UniqueName: \"kubernetes.io/projected/054b0c8e-37b7-4837-8b3d-5a6a422ed171-kube-api-access-9wzxf\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:40:16.228352 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.228350    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054b0c8e-37b7-4837-8b3d-5a6a422ed171-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:40:16.228473 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.228359    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4511df-478e-4cda-9b83-e6fac0490d3f-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:40:16.228473 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.228366    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2drzd\" (UniqueName: \"kubernetes.io/projected/92de2058-3d2c-4b20-b78f-cff131643350-kube-api-access-2drzd\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:40:16.228473 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.228374    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7crb2\" (UniqueName: \"kubernetes.io/projected/cf4511df-478e-4cda-9b83-e6fac0490d3f-kube-api-access-7crb2\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:40:16.228473 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.228382    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92de2058-3d2c-4b20-b78f-cff131643350-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Jul 31 02:40:16.350071 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.350051    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:16.889642 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.889624    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"
Jul 31 02:40:16.889973 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.889626    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9" event={"ID":"92de2058-3d2c-4b20-b78f-cff131643350","Type":"ContainerDied","Data":"f0971026d28865825060d24fb958a4b463b83f0ac5c5d48260d548dcd564316d"}
Jul 31 02:40:16.889973 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.889742    2556 scope.go:117] "RemoveContainer" containerID="8a81cbcec778f8b8c1eb65db5b8fef316dfc0adb3cf4d87ee9fc558c26d873a5"
Jul 31 02:40:16.891017 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.891001    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"
Jul 31 02:40:16.891084 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.891026    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc" event={"ID":"054b0c8e-37b7-4837-8b3d-5a6a422ed171","Type":"ContainerDied","Data":"b4a5a7a7fa1a7ed792267ce1cc42879bdabb855a691d18e980356d891e078c16"}
Jul 31 02:40:16.892306 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.892289    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n" event={"ID":"cf4511df-478e-4cda-9b83-e6fac0490d3f","Type":"ContainerDied","Data":"3b5feeb2a88643b3ba45b1f81da3822b7eda83b943f65d226b7fc11bd6f59c47"}
Jul 31 02:40:16.892384 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.892321    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"
Jul 31 02:40:16.900744 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.900726    2556 scope.go:117] "RemoveContainer" containerID="4e301479f08431bec62583bd0cf6b834e55d5ace4e08fac1c8f82329a93d4256"
Jul 31 02:40:16.908086 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.908070    2556 scope.go:117] "RemoveContainer" containerID="88602f035ca93b90967a734a4b3f8a1fa296967fd6d174d34e58e9b5c3fba63e"
Jul 31 02:40:16.916132 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.916114    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"]
Jul 31 02:40:16.919154 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.919135    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-01a44c-b4bf59474-q4cc9"]
Jul 31 02:40:16.958492 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.958472    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"]
Jul 31 02:40:16.963052 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.963034    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-01a44c-7b9878d54d-x8qdc"]
Jul 31 02:40:16.981957 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.981940    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"]
Jul 31 02:40:16.988829 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:16.988810    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-01a44c-7587778796-4hl5n"]
Jul 31 02:40:18.151673 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:18.151645    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="054b0c8e-37b7-4837-8b3d-5a6a422ed171" path="/var/lib/kubelet/pods/054b0c8e-37b7-4837-8b3d-5a6a422ed171/volumes"
Jul 31 02:40:18.152013 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:18.151951    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92de2058-3d2c-4b20-b78f-cff131643350" path="/var/lib/kubelet/pods/92de2058-3d2c-4b20-b78f-cff131643350/volumes"
Jul 31 02:40:18.152229 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:18.152217    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf4511df-478e-4cda-9b83-e6fac0490d3f" path="/var/lib/kubelet/pods/cf4511df-478e-4cda-9b83-e6fac0490d3f/volumes"
Jul 31 02:40:22.466596 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:22.466547    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:22.572831 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:22.572808    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:22.742188 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:22.742127    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:22.981200 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:22.981175    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:23.082639 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:23.082615    2556 scope.go:117] "RemoveContainer" containerID="0233e38de75588bdb3acf1df1a99a0ccd41f6ffe15de6feac6992af2abdaaa41"
Jul 31 02:40:23.090200 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:23.090180    2556 scope.go:117] "RemoveContainer" containerID="7b2c0b1e7825b98593e6808254fbb9b026cb7662627205fa466e4755f152eb42"
Jul 31 02:40:29.507618 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:29.507591    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7745c54f68-8lpz7_a44fce3b-698b-4e2d-8222-098849a2b9be/manager/0.log"
Jul 31 02:40:30.241934 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:30.241913    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77df9dc58b-l7b9g_14d969da-0f92-4e5e-ab34-f2acc6e470ff/copy-manifests/0.log"
Jul 31 02:40:30.251168 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:30.251152    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77df9dc58b-l7b9g_14d969da-0f92-4e5e-ab34-f2acc6e470ff/manager/0.log"
Jul 31 02:40:30.619835 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:30.619817    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-77f76cbbbd-27pnj_254c4573-cac4-45e1-9368-11a24bb2841e/maas-api/0.log"
Jul 31 02:40:30.745137 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:30.745111    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29757720-g6gfm_ba9384ff-bddc-484e-916d-1745b71611d1/cleanup/0.log"
Jul 31 02:40:30.864367 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:30.864340    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29757735-zfcjm_e74e4016-055d-48be-8dca-a8e7e0d024fb/cleanup/0.log"
Jul 31 02:40:30.991277 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:30.991226    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29757750-cmxtx_7eb499d2-f7d7-44df-a0f0-db524cb64594/cleanup/0.log"
Jul 31 02:40:31.115980 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:31.115960    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-dqzch_8ffafa2b-6ae5-4a3d-90f3-469e80316950/postgres/0.log"
Jul 31 02:40:33.364507 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:33.364474    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fjkxr_af296a8a-d9e7-4047-bef6-ad81aab47ddd/discovery/0.log"
Jul 31 02:40:33.914870 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:33.914849    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wfhq9_e66c5894-f7c2-4e5a-8a9c-ecd308c986df/payload-pre-processing/0.log"
Jul 31 02:40:34.039660 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:34.039643    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-l2mkg_fa01e1a2-7436-45ba-b058-346e7534e8ec/payload-processing/0.log"
Jul 31 02:40:34.560128 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:34.560110    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jbmx5_cf7bcc7e-32cc-47ff-b5e1-fd5000676ec8/main/0.log"
Jul 31 02:40:41.639851 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:41.639819    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:41.868944 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:41.868918    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log" limit=1024
Jul 31 02:40:45.573141 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:45.573114    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dnk4v_32e17105-e7bb-4ad2-8ef5-77660ff9a429/global-pull-secret-syncer/0.log"
Jul 31 02:40:45.807602 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:45.807572    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-88pgt_e26e8e31-84dc-4178-9f3d-172319dd06cb/konnectivity-agent/0.log"
Jul 31 02:40:46.148515 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:46.148480    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-142.ec2.internal_f923c99785cedeeb9d5a1e0081b46a58/haproxy/0.log"
Jul 31 02:40:53.556527 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:53.556499    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dxbfb_2b86f0a5-bacf-430e-b61b-21769d948195/node-exporter/0.log"
Jul 31 02:40:53.608219 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:53.608194    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dxbfb_2b86f0a5-bacf-430e-b61b-21769d948195/kube-rbac-proxy/0.log"
Jul 31 02:40:53.650676 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:53.650661    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dxbfb_2b86f0a5-bacf-430e-b61b-21769d948195/init-textfile/0.log"
Jul 31 02:40:53.947093 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:53.947041    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/prometheus/0.log"
Jul 31 02:40:53.998809 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:53.998790    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/config-reloader/0.log"
Jul 31 02:40:54.048389 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.048371    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/thanos-sidecar/0.log"
Jul 31 02:40:54.100282 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.100261    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/kube-rbac-proxy-web/0.log"
Jul 31 02:40:54.147488 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.147474    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/kube-rbac-proxy/0.log"
Jul 31 02:40:54.195927 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.195910    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/kube-rbac-proxy-thanos/0.log"
Jul 31 02:40:54.275352 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.275302    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2baf2933-1533-4bc7-8243-c1b7e6a011ad/init-config-reloader/0.log"
Jul 31 02:40:54.436831 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.436813    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"]
Jul 31 02:40:54.437100 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.437088    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf4511df-478e-4cda-9b83-e6fac0490d3f" containerName="payload-pre-processing"
Jul 31 02:40:54.437155 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.437101    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf4511df-478e-4cda-9b83-e6fac0490d3f" containerName="payload-pre-processing"
Jul 31 02:40:54.437155 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.437112    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054b0c8e-37b7-4837-8b3d-5a6a422ed171" containerName="maas-api"
Jul 31 02:40:54.437155 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.437118    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="054b0c8e-37b7-4837-8b3d-5a6a422ed171" containerName="maas-api"
Jul 31 02:40:54.437155 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.437125    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92de2058-3d2c-4b20-b78f-cff131643350" containerName="payload-processing"
Jul 31 02:40:54.437155 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.437130    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="92de2058-3d2c-4b20-b78f-cff131643350" containerName="payload-processing"
Jul 31 02:40:54.437155 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.437146    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="293bbde2-3cda-4954-84f1-6a3a11e028a6" containerName="revoke-keys"
Jul 31 02:40:54.437155 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.437151    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="293bbde2-3cda-4954-84f1-6a3a11e028a6" containerName="revoke-keys"
Jul 31 02:40:54.437361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.437194    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="92de2058-3d2c-4b20-b78f-cff131643350" containerName="payload-processing"
Jul 31 02:40:54.437361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.437201    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="293bbde2-3cda-4954-84f1-6a3a11e028a6" containerName="revoke-keys"
Jul 31 02:40:54.437361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.437207    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf4511df-478e-4cda-9b83-e6fac0490d3f" containerName="payload-pre-processing"
Jul 31 02:40:54.437361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.437216    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="054b0c8e-37b7-4837-8b3d-5a6a422ed171" containerName="maas-api"
Jul 31 02:40:54.440156 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.440143    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.445929 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.445749    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tsc55\"/\"openshift-service-ca.crt\""
Jul 31 02:40:54.445929 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.445780    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tsc55\"/\"kube-root-ca.crt\""
Jul 31 02:40:54.446431 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.446418    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tsc55\"/\"default-dockercfg-p9l6w\""
Jul 31 02:40:54.464060 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.464031    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"]
Jul 31 02:40:54.490796 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.490779    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-podres\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.490875 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.490807    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-lib-modules\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.490875 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.490830    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-sys\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.490944 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.490868    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-proc\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.490944 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.490920    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr5wx\" (UniqueName: \"kubernetes.io/projected/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-kube-api-access-tr5wx\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.499094 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.499074    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-lgpmv_b433e829-5435-43f5-912b-1b6a65e29be4/prometheus-operator-admission-webhook/0.log"
Jul 31 02:40:54.591984 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.591962    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr5wx\" (UniqueName: \"kubernetes.io/projected/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-kube-api-access-tr5wx\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.592289 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.591996    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-podres\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.592289 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.592023    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-lib-modules\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.592289 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.592048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-sys\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.592289 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.592073    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-proc\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.592289 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.592136    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-proc\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.592289 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.592154    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-podres\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.592289 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.592160    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-lib-modules\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.592289 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.592169    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-sys\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.603336 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.603308    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr5wx\" (UniqueName: \"kubernetes.io/projected/5f2ece0b-2573-4bae-bb81-52a9c81f7e03-kube-api-access-tr5wx\") pod \"perf-node-gather-daemonset-fvn45\" (UID: \"5f2ece0b-2573-4bae-bb81-52a9c81f7e03\") " pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.749347 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.749324    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:54.888260 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:54.888228    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"]
Jul 31 02:40:54.889515 ip-10-0-141-142 kubenswrapper[2556]: W0731 02:40:54.889484    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5f2ece0b_2573_4bae_bb81_52a9c81f7e03.slice/crio-c8704f6178673fe65ba0b73cfad8cfe27efb8abf5108d8310f94529e1292ba34 WatchSource:0}: Error finding container c8704f6178673fe65ba0b73cfad8cfe27efb8abf5108d8310f94529e1292ba34: Status 404 returned error can't find the container with id c8704f6178673fe65ba0b73cfad8cfe27efb8abf5108d8310f94529e1292ba34
Jul 31 02:40:55.046619 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:55.046596    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45" event={"ID":"5f2ece0b-2573-4bae-bb81-52a9c81f7e03","Type":"ContainerStarted","Data":"cd2b726514de7303a0a74dfd81ebe22b1e82bc7c24a3c9e9a5b96678a557c4bc"}
Jul 31 02:40:55.046711 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:55.046642    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45" event={"ID":"5f2ece0b-2573-4bae-bb81-52a9c81f7e03","Type":"ContainerStarted","Data":"c8704f6178673fe65ba0b73cfad8cfe27efb8abf5108d8310f94529e1292ba34"}
Jul 31 02:40:55.046711 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:55.046671    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:40:57.489153 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:57.489121    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-w6ktx_b1ea40d4-e2d3-46b7-b4a6-9d54bfefdb0f/download-server/0.log"
Jul 31 02:40:59.154827 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:59.154802    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8q5rf_90bc69db-3427-4022-83ac-b134a6898421/dns/0.log"
Jul 31 02:40:59.181094 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:59.181073    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8q5rf_90bc69db-3427-4022-83ac-b134a6898421/kube-rbac-proxy/0.log"
Jul 31 02:40:59.402201 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:40:59.402177    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tqppw_8525e366-5355-4406-8e20-88f336bf1e44/dns-node-resolver/0.log"
Jul 31 02:41:00.023067 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:00.023044    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lprjl_d5c6c5a5-2a0c-48e2-b968-93837c492672/node-ca/0.log"
Jul 31 02:41:01.060403 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:01.060380    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45"
Jul 31 02:41:01.085910 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:01.085871    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tsc55/perf-node-gather-daemonset-fvn45" podStartSLOduration=7.085845706 podStartE2EDuration="7.085845706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 02:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 02:40:55.073383438 +0000 UTC m=+3813.439129073" watchObservedRunningTime="2026-07-31 02:41:01.085845706 +0000 UTC m=+3819.451591331"
Jul 31 02:41:01.253234 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:01.253205    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fjkxr_af296a8a-d9e7-4047-bef6-ad81aab47ddd/discovery/0.log"
Jul 31 02:41:01.394432 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:01.394372    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wfhq9_e66c5894-f7c2-4e5a-8a9c-ecd308c986df/payload-pre-processing/0.log"
Jul 31 02:41:01.425829 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:01.425811    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-l2mkg_fa01e1a2-7436-45ba-b058-346e7534e8ec/payload-processing/0.log"
Jul 31 02:41:02.436382 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:02.436357    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s6rd8_144cf673-df86-42f6-9d5a-925a3fb98dec/serve-healthcheck-canary/0.log"
Jul 31 02:41:03.219462 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:03.219438    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k4jqp_926bcd5f-f88b-44fb-b4f4-acee33c2d448/kube-rbac-proxy/0.log"
Jul 31 02:41:03.261311 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:03.261293    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k4jqp_926bcd5f-f88b-44fb-b4f4-acee33c2d448/exporter/0.log"
Jul 31 02:41:03.292175 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:03.292157    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k4jqp_926bcd5f-f88b-44fb-b4f4-acee33c2d448/extractor/0.log"
Jul 31 02:41:05.644361 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:05.644336    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fjkxr_af296a8a-d9e7-4047-bef6-ad81aab47ddd/discovery/0.log"
Jul 31 02:41:05.754647 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:05.754629    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wfhq9_e66c5894-f7c2-4e5a-8a9c-ecd308c986df/payload-pre-processing/0.log"
Jul 31 02:41:05.780245 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:05.780227    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-l2mkg_fa01e1a2-7436-45ba-b058-346e7534e8ec/payload-processing/0.log"
Jul 31 02:41:06.697950 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:06.697919    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7745c54f68-8lpz7_a44fce3b-698b-4e2d-8222-098849a2b9be/manager/0.log"
Jul 31 02:41:07.246910 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:07.246885    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77df9dc58b-l7b9g_14d969da-0f92-4e5e-ab34-f2acc6e470ff/manager/0.log"
Jul 31 02:41:07.302674 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:07.302655    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77df9dc58b-l7b9g_14d969da-0f92-4e5e-ab34-f2acc6e470ff/copy-manifests/0.log"
Jul 31 02:41:07.425190 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:07.425158    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fjkxr_af296a8a-d9e7-4047-bef6-ad81aab47ddd/discovery/0.log"
Jul 31 02:41:07.574402 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:07.574365    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wfhq9_e66c5894-f7c2-4e5a-8a9c-ecd308c986df/payload-pre-processing/0.log"
Jul 31 02:41:07.604353 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:07.604333    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-l2mkg_fa01e1a2-7436-45ba-b058-346e7534e8ec/payload-processing/0.log"
Jul 31 02:41:08.671397 ip-10-0-141-142 kubenswrapper[2556]: I0731 02:41:08.671367    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8657cf6695-45dbm_24cbb706-1704-4a27-a806-8e753e3bdc10/manager/0.log"